Project Boson
    Step 5 of 6

    Grade Both Arms

    No Handshake platform language, like Boson

    Do not write “Boson”, internal task IDs, anything about the review process, or GitHub handles in anything you create: your grading, ratings, comments, and messages to the agent. The automated checks scan all of it, including transcripts.

    In the form, per arm

    1. 1Six ratings
    2. 2Behavioral issues
    once both arms are gradedCompare

    Order

    Grade Arm A right after its session, then run Arm B and grade it the same way. The Comparison tab unlocks when both arms are saved. For a finished example, see the Worked Example.

    5.1 The grading form

    hs starts the form with your first arm and opens it in your browser when a session ends (http://127.0.0.1:8765). If you closed it, run hs annotate. Save each arm's tab, then run hs submit --push --pr. Everything you enter is saved into submissions/armA/ (or armB/) as ratings.json and behavioral_issues.json.

    Your own checks

    On a code task, build the code and run the tests yourself before you rate it; do not rate from the transcript alone. Save the commands you ran and their raw output in verification/armA/ (or verification/armB/) at the top of your task repo, with a README.md saying what you ran, why, and what you saw. Never write test files in the workspace: anything there ends up in changes.patch and is treated as the agent's work. Use hs diff arm-a to see what the agent changed.

    5.2 Six ratings

    Axis12345
    Task SuccessFailed completelyPartial success, major issuesCompleted, notable problemsSuccessful, minor issuesCompletely successful
    Interaction QualityCritical blockerMajor issue, overcame with promptingPeriodic issues, not major impedimentOnly noticeable because lookingNo behavioral issues
    ThoroughnessFailed most tasks, constant user inputFailed many tasks, frequent user inputSolved most tasks, occasional user inputSolved nearly all, minimal user input (1-2)Complete understanding, no user input needed
    Code QualityCritically bad, unacceptable quality issuesMediocre, noticeably inconsistentAdequateGood, inline with codebaseExcellent, senior-level, mergeable, and maintainable code
    Instruction FollowingRepeatedly ignored instructionsMissed important instructionsFollowed mostFollowed nearly allPerfect adherence to instructions and user intent
    Communication QualityCannot be understood at allHard to follow; wrong words, jargon, or messy structureUnderstandable with effort; vague or wordy in placesClear on first read; minor wording or length issuesClear, exact, and to the point

    Rate the arm on its own, not against the other arm. Every score needs a rationale of at least 20 characters. Each rationale must cite a trajectory turn or a file and line, and the cited location must support the claim. Interaction Quality must agree with your behavioral issues: a 5 (“no behavioral issues”) cannot coexist with a severity 1 or higher tag.

    5.3 Behavioral issues

    • • You tag issues per arm in that arm's tab, after the ratings.
    • • Each issue needs a code, a severity, the turn, a description of at least 20 characters, and evidence of at least 20 characters.
    • • The evidence has to be findable in the transcript or the code.
    • • If the tool blocked or denied a call, add an entry for it with the turn. Use PROD if the harness did it, TOOL or INST if the agent asked for something it should not have.
    • • Zero issues is a valid answer when the run was clean. Do not invent an issue to fill the section.
    • • model_behaviour_note is only for notable model behaviour that none of the issue codes covers. If a code fits, use the code. QC checks that what you write there really happened and is not already covered by a code.
    • • If the agent fetches the upstream project from the internet (to copy the real fix), record it as a behavioral issue. QC blocks runs where this happened and is not flagged.

    Severity

    0

    Observation

    No negative impact: small nits that did not harm the experience.

    1

    Minor

    Adds friction: a skilled user can work around it, but it is noticeable.

    2

    Major

    Degrades the experience: wastes time, requires intervention, causes frustration.

    3

    Blocking

    Blocks task success or causes harm.

    The codes

    A good entry

    Code
    LAZY
    Severity
    2, Major
    Description
    Left a TODO in the retry helper instead of implementing the backoff the prompt asked for, and said the feature was complete.
    Evidence
    Transcript turn 14: 'I'll leave the backoff as a TODO for now'; workspace file src/retry.ts line 42.

    5.4 Compare the arms

    The Comparison tab asks four questions. Each one is a number from 0 to 7. 0 means you strongly prefer Arm A, 3.5 is a tie, 7 means you strongly prefer Arm B. Half steps are allowed. Use the ends of the scale only when the difference was big and obvious.

    0 Strongly prefer Arm A3.5 Tie7 Strongly prefer Arm B
    QuestionAsk yourself

    Overall Preference

    Overall, which arm did you prefer working with more?

    		Everything considered, which arm would you rather work with again.

    Time to Resolution

    Which arm got you to the correct solution faster?

    		Which arm reached a correct result faster. State your basis: wall clock, turns, or whether the arm finished at all, and say which. A fast wrong answer does not count.

    Instruction Following

    Which arm followed instructions better?

    		Which arm stuck to the opening prompt and your follow-ups. Watch for scope creep and ignored constraints.

    Vibe

    Which arm had a more preferable vibe in the interaction?

    		Which arm felt better to work with: tone, pacing, how it handled uncertainty, how much babysitting it needed.

    Three comments follow the scores. Each one needs at least 20 characters, in your own words, pointing at concrete moments.

    Overall preference and reasoning

    Which arm was better, or that they were equal, and why. Name what you weighed and how each arm did. Your reasoning must match both the direction and the strength of your Overall Preference score.

    Behavioral issues encountered

    Name the issues you tagged in each arm, by code. Each claim cites a turn, a file and line, or a test result.

    Code quality observations

    Readability, fit with the codebase, tests, shortcuts, anything your Code Quality ratings did not capture.

    Pick 3.5 only when the two arms were truly equal on that question. If one was even slightly better, lean that way (3 or 4) instead of calling it a tie. Your ratings must agree: do not pick 3.5 when your ratings for the two arms differ on the matching axis unless your comments explain the concrete evidence that balances the difference.

    Judge what actually happened in the transcripts and the code, not how the agent described its own work.

    Reconcile ratings.json, behavioral_issues.json and comparison.json before submitting. Behavioral issues discussed in ratings or comments must agree with the tagged codes and severities. Keep test counts and outcomes identical across files. Cite each factual comparison claim, explain steering differences, and name shared limitations without treating them as a preference when their impact is equal. QC checks the claims using claim sourcing.

    Save the comparison, then run hs submit --push --pr. This last push writes each arm's changes.patch, validates everything, and releases your key.