Automated Checks

How QC works

QC combines static validation with model-based review of each arm, consistency across the evaluation files, and sourcing of individual claims. Checks run on every push, after each of the five steps, and comment on the pull request. A reviewer assesses the findings against the record. Passing local validation does not establish that the grading is accurate or that the submission will be approved.

Static validation

Required files and JSON schemas are checked: settings.json, run.json, ratings.json, behavioral_issues.json, and the session artifacts for each arm, plus comparison.json for the pair. Each rating is an integer from 1 to 5; the four preference scores range from 0 to 7.

Checks also examine assigned models, matching opening prompts, arm order, distinct session identifiers, reused sessions, and duplicate evaluation files. Identical scores alone do not establish duplication; each arm’s evidence and rationale must come from its own record. Keep harness-generated settings, timestamps, task metadata, and transcripts intact. Keep the experiment setup out of agent sessions.

Separate arm reviews

Each arm is reviewed independently for interaction-mode compliance, an end-to-end problem-solving record, setup leakage, environmental events, and agreement between the trajectory and patch. Logs must be complete, describe the same session, and include subagent logs when helpers ran. A must finish before B starts.

A run cut off by an API error, connection loss, or budget cap cannot be treated as a completed attempt merely because the grading discloses it. Rerun the arm and give the reason when hs asks. A new Arm A means Arm B runs again too. A run stopped by the 2h30m limit is re-run if the time ran out while the agent was waiting on you or your machine was asleep; if the agent was working the whole time, it is a genuine result and is graded as it is. A genuine model failure is still an evaluation outcome; do not rerun to obtain a better answer. Record missing toolchains, denied calls, failed builds, and recovery with turn numbers. If Arm A hits a missing tool, add it to envs/tasks/<prompt_id>.dockerfile and restart Arm A from scratch; during Arm B, finish the arm as it is and say so in the comparison comments, citing the turns.

Consistency across the evaluation files

Six ratings, behavioral issues, and comparison comments must agree. An issue described in a rating or comparison must be reflected in the behavioral record when it is a behavioral issue. Use the same codes, severities, test counts, and factual outcomes throughout. Reviewers also check for omitted issues in the trajectory and patch, including runs marked as having no issues.

Retain the current scoring constraints: a severity 1 or higher behavioral tag rules out 5 for Interaction Quality. Explain the impact rather than calculating scores from a failure count. A comparison tie against different per-arm ratings requires a specific explanation of the evidence that balances the difference.

Name shared limitations once and explain why they do not move the preference if they affected both arms equally. Document steering differences and their effect on fairness. Vibe concerns tone, honesty, and handling of the interaction; it is not a second Task Success score. State the basis of Time to Resolution explicitly.

Claim sourcing

Every factual claim is checked against its cited evidence, including claims in rating rationales and comparison comments. Cite a trajectory turn, a patched file and line identifiable in changes.patch, or a command and its actual output. Distinguish patched-file line numbers from line numbers in the patch artifact. External checks must be clearly identified as your own; they need not appear in the agent’s session. Cite only the trajectory and the patch. Do not point to run.json or any other harness file in your grading, ratings, or comments.

For independent tests, preserve the test source and output. Explain the tested cases, expected behavior, exact commands, environment, submitted patch, and observed results. Mark performance, memory, and scale claims as measured or inferred from code. A bare statement that tests passed does not establish what was tested.

Resolving findings

Dead or truncated runs, wrong model attribution, setup leakage, material prompt or steering mismatches, missing required artifacts, and citations that do not support their claimed location prevent acceptance until resolved. Fixable findings include citation mistakes, recoverable log discrepancies corroborated by other records, and partial verification limited by an external dependency. Explain what remains unverified.

Policy clarifications still pending

The updated audit checklist contains conflicting or provisional instructions on the points below. Until Ops confirms them, retain the existing workflow and ask #boson-tasking before making an exception.

• Autonomous replies: some sections allow narrow answers to agent questions; others prohibit all follow-ups. Do not introduce this exception without confirmation.
• Interactive steering: the checklist alternates between identical follow-ups and comparable guidance. Retain the same-follow-up rule and document deviations and their effect.
• Verification artifacts: save your own checks and raw output in verification/armA/ and verification/armB/, with a README.md saying what you ran, why, and what you saw.
• Time to Resolution: the checklist mentions both finishing first and a stated basis such as elapsed time, turns, or completion. Continue stating time to a correct result and its basis; do not compare absolute finish timestamps because A runs first.
• Enforcement: checks run on every push after each of the five steps. Check IDs and fail/warn thresholds need confirmation. The audit checklist alone does not establish deployed behavior.