Project Boson
    Step 4 of 6

    Run Arm A and Arm B

    No Handshake platform language, like Boson

    Do not write “Boson”, internal task IDs, anything about the review process, or GitHub handles in anything you create: your grading, ratings, comments, and messages to the agent. The automated checks scan all of it, including transcripts.

    Push after every step

    The task is five steps: run Arm A, grade Arm A, run Arm B, grade Arm B, compare. After each one run hs submit --push --pr. The first push opens the pull request; later pushes add to it. The automated checks comment on the pull request after every push, usually within 10 minutes. Wait for that comment before the next step; if it sends a run back, redo that run before going on.

    Five pushes, checked one at a time

    PushStepWhat QC checks on this push
    1Run Arm AArm A's run: transcript, run.json, settings.json, the session ran on the pinned config.
    2Grade Arm AArm A's ratings and behavioral issues are complete and consistent with the transcript.
    3Run Arm BArm B's run, and that it came after Arm A on the same image.
    4Grade Arm BArm B's ratings and behavioral issues.
    5CompareThe comparison, both changes.patch files, and the whole submission end to end.

    A step you have not reached yet is not an error; a step you have pushed must be complete and in order. If QC sends a run back, redo that run and everything built on it (its grading, and Arm B if it was Arm A). hs submit runs the same validator as CI and refuses to push if anything is missing or still says <fill in>.

    Arm A is hs arm-a. Arm B is hs arm-b. Always in that order; the tool refuses Arm B until Arm A has run.

    Order

    Run Arm A, grade Arm A (Step 5), then run Arm B and grade it the same way. The Comparison tab unlocks when both arms are saved.

    Start Arm A

    hs arm-a

    What happens

    1. The first run builds the Docker image. It takes a few minutes.
    2. The banner names the arm. There are no theme, trust, or key prompts; the tool answers them.
    3. The opening prompt is sent as the first message. In an Interactive task it is also placed on your clipboard for reference. Never send anything before the agent's first reply.
    4. The workspace opens in your editor, and the grading form starts in the background at http://127.0.0.1:8765.
    5. The session ends on its own after 2 hours 30 minutes if you have not ended it.

    When the session ends, run hs submit --push --pr.

    End the session

    Press Ctrl-C when the agent is done, when it has clearly given up or is looping, or when the time limit ends it (see The 2h30m session limit). The session is exported and collected into submissions/armA/ on its own.

    There is no resume: every run starts from scratch. If a run has to be redone (crash, lost connection, API error), run the arm again. hs asks why and records your answer, and keeps the earlier run under submissions/previous-runs/. A new Arm A means Arm B runs again too. Never rerun an arm to give the agent another try.

    Keep every run: never delete anything in submissions/previous-runs/. QC checks each re-run has its own folder with its transcript, run.json, settings.json, changes.patch and reason. Grade only the latest run of each arm, the one in submissions/armA/ and submissions/armB/.

    Interrupted run or model failure

    Do not submit a trajectory cut off by an API exception, connection loss, or budget cap as a completed attempt. Rerun the arm and give the reason when hs asks; if recovery fails, ask #boson-tasking for rerun guidance. Describe the affected turns and the recovery in your grading evidence. A poor solution or genuine model failure is an outcome, not permission to rerun for a better answer. If the agent hits a missing tool during Arm A, stop, add it to the task dockerfile (Step 3), and restart Arm A from scratch. During Arm B, finish the arm as it is and say so in the comparison comments, citing the turns.

    The 2h30m session limit

    • Each arm's session is capped at 2 hours 30 minutes, counted from the moment the session starts.
    • The clock is wall-clock time: time your laptop is asleep or closed still counts, and idle time counts.
    • When the limit is hit, the session is stopped and the run is recorded as stopped by the time limit.

    A stopped session cannot be resumed. What you do next depends on why the clock ran out:

    You went idle

    You went idle: re-run the arm. If the time ran out while the agent was waiting on you (for a reply, a follow-up or an approval), or while your laptop was asleep or closed, the run cannot be submitted. Re-run that arm from scratch; the killed run is kept under submissions/previous-runs/ with your reason.

    The agent was working the whole time

    The agent was working the whole time: submit it. If the agent was still actively working when the limit hit and genuinely could not solve the task in the time allowed, that is a real result. Grade the run as it is, and say in the comparison comments that the time limit ended it, citing the last turns.

    If Arm A is re-run, Arm B has to be run again after it as well.

    Plan your time so the agent can finish: do not start an arm unless you can stay with it for the full session.

    Record issues as they happen

    Run hs issue arm-a (or hs issue arm-b) while the arm runs. It asks for the code, severity, what happened and the evidence; the grading form shows it pre-filled. You can still add issues in the form afterwards.

    Start Arm B

    hs arm-b

    It refuses to start until Arm A has run. Then run hs submit --push --pr.

    Same as Arm A, with the same steering. If the number of follow-ups differs, explain why in the comparison comments, citing the turns. Nothing carries over between the arms.

    Do not

    • Edit files in the workspace during a run.
    • Use /model, /effort, extra flags, or HS_* environment variables.
    • Add instruction files, skills, or hooks to the workspace or the agent's config.
    • Open the workspace in any AI tool during a run. After the run, only the assistant, for facts and tests.
    • Try to identify which model is behind an arm, or look up how the arms are configured.
    • Put "Arm A", "Arm B", hs commands, or the words "grading form" into the agent session.
    • Change the task dockerfile once Arm A has started, or install packages into the running container.