On This Page

Reviewer Guidelines

Read everything

Check every piece of text the fellow wrote, not a skim of it. Check every evidence citation, every rating, and every comparison axis, and check for generic AI-written prose everywhere the fellow wrote. If you did not verify it yourself against the raw transcript or the diff, you have not checked it.

Fail fast on dealbreakers

Identify whether the comparison is unusable or the record can be corrected. Request specific changes; do not treat every recoverable artifact defect as grounds to discard the run.

Generic AI-written prose in ratings.json, behavioral_issues.json, or comparison.json. If a sentence would read the same on any submission ("the submission demonstrates comprehensive coverage", "the agent handled the task effectively") it is not evidence, and it shows the fellow did not engage with this run. Check every rationale field.
The opening prompt is wrong, or metadata was altered.
Material steering mismatches that bias the comparison. Autonomous reply exceptions and the precise interactive parity rule are pending clarification; escalate ambiguous cases rather than inventing an exception. Record the actual messages and their effect.
Passive review on a code task: the fellow graded an implementation task by reading the transcript without compiling or running the tests locally. Request verification. If an external dependency prevents only part of the checks, distinguish the blocked portion from what was verified and assess whether the remaining evidence supports the ratings.

Confirm the artifacts are real

Both arms have transcript/session.jsonl and transcript/trajectory.json, and both are non-empty.
changes.patch exists for both arms and, when non-empty, starts with a real diff header. A file that exists but is not a diff does not count.
run.json and settings.json are consistent with the arm: the model alias and the effort level in settings.json are the ones the tool pins for that arm, and the two arms differ from each other in the expected way. If the two arms show the same effort, or swapped values, with no note explaining it, that is an undisclosed confound stacked on top of everything else being compared.
Repository hygiene: no files outside the submission structure were committed. Stray verification.md, .gitkeep, extra notes, or scratch scripts in the arm folders are a defect under the current submission structure. Independent test artifacts and raw output go inverification/armA/ and verification/armB/ with a README.md.

Read the task before either arm

Check the task type. Code, bug fix, or implementation tasks require real code changes and reproducible test execution. REQS tasks (architecture spec, ADR, RFC) asked for scoping or documentation with no code, and are held to the documentation standard in step 4.

Verify the code and tests yourself, do not just read the logs

For each arm, in order, and for every item. Not a sample.

Code tasks

Check that the fellow compiled and ran the code locally on a clean checkout.
No transcript echoing: quoting or summarising the agent's own output from session.jsonl (for example quoting "142 passed, 1 warning") is not independent verification. If the evidence has no local build or test commands with their output, do not approve. Request changes.
For optimizer, query, or logic bugs, check that the fellow ran differential on and off checks (for example PRAGMA disable_optimizer).

REQS and architecture spec tasks

When the opening prompt forbade writing code and asked for a technical spec or ADR, compilation is not required.
Verification must rest on line-level citations in changes.patch showing the fellow checked runtime boundary consistency, threading models, and interface contracts.

Evidence strings in ratings.json

Evidence must cite concrete proof: the exact command run (for example pytest, go test ./..., or ctest -R), verified assertion counts (for example 192 assertions passed), and exact turn locators (Turn X) from trajectory.json.
Raw session.jsonl line numbers are not acceptable; cite the turn number.
Classify sourced claims as present, slip (real fact, wrong location), overstated, contradicted, absent, or unverifiable. Verify the actual content at the cited location. A wrong locator must be corrected; an unsupported claim may also require changing the score. Use patched-file line numbers identifiable in changes.patch, and label patch-artifact line numbers explicitly.
No phantom logs: if an item asks for install logs (for example "Setting up" lines) or verification runs, the actual command and output must be available in the evidence or approved verification artifacts. External checks need not appear in the agent trajectory. Establish who ran them, on which submitted patch, with what environment and test source. A missing or inaccessible result requires correction; do not confuse independent execution with an invented session result.

Diff audit

Check changes.patch against what the transcript shows being edited. A Task Success rating above 2 next to an empty or near-empty patch, when the task required a code change, is a blocker even if the transcript narrates a fix.

Slop scan

Watch for generic AI-written prose inside the evidence strings themselves, not only in the top-level rationale fields.

Cross-check every rating and every behavioral tag against what you read

No interaction_quality of 5 ("no behavioral issues") next to a severity 1 or higher tag in behavioral_issues.json.
Every tag code is one of the 13: PROD, INST, OVERENG, TOOL, LAZY, VERIFY, FALSE, ROOT, DESTRUCT, FILE, HALLUC, DOCS, VERBOSE.
Turn provenance is mandatory: every entry in behavioral_issues.json must include a non-empty turns list ([X, Y]) pointing at real turn numbers in trajectory.json. An empty turns list, or a quote of the agent with no turn number, is an automatic send-back.
Audit clean runs: if behavioral_issues.json is empty, check the trajectory and patch for omitted issues. An empty file is not by itself evidence of misconduct.
Watch for the opposite failure: a tag or rating that accuses the agent of something the transcript does not support.
If both arms' rationales read alike on more than one axis, check whether they describe the same real facts or one was copied without being re-derived from that arm's own run.

Check the two arms against each other

order_in_pair: the arm marked 1 has the earlier real start_utc of the two.
Steering parity: inspect the content and effect of follow-ups, not just their count. Record differences and assess fairness against the confirmed policy. A note claiming no bias is not proof. The updated checklist’s identical-versus-comparable wording is awaiting Ops clarification.
Each arm's declared start and end window matches its transcript's real first and last timestamp. B starts after A ends. Confirm distinct session IDs, no reused trajectory, matching raw and exported logs, and complete helper logs.

Check every comparison axis

Four axes, 0 to 7: Overall Preference, Time to Resolution, Instruction Following, Vibe. Check each one on its own.
Consistency across scales: the 0 to 7 comparison scores must agree with the 1 to 5 per-arm ratings. If the per-arm Instruction Following ratings put Arm B above Arm A, the comparison score on that axis cannot favor Arm A (below 3.5).
Grounding: every number matches its own written reasoning.
Reject vague reasoning on sight ("Arm B felt faster"). Every claim needs a cited turn or diff line, checked against the trajectory the same way you checked grading evidence.
Reconcile the three evaluation files: ratings, behavioral issues, and comparison. The comments use the same behavioral codes and severities, and factual claims and test counts match the arm files. Source every claim. Issues raised in ratings or comments must also appear in the behavioral record when applicable. Explain shared limitations, steering differences, and ties against differing ratings.

QC includes static validation, independent arm reviews, cross-file consistency, and claim sourcing. An unusable run, wrong model, setup leak, or material prompt mismatch prevents acceptance. Recoverable log discrepancies and partial verification blocked by external dependencies can be corrected or assessed on corroborating evidence. Consult the pending policy clarifications for steering exceptions, artifact paths, environment changes, and enforcement thresholds.

Make the call and score the review

Never default to 5. Only 4 and 5 are approved; everything else goes back to the fellow.

Score	Decision	What it takes
5 Exemplary	Approve	The fellow built the workspaces locally, ran independent differential suites, and verified test assertions directly. Exact Turn X citations verified in `trajectory.json`. Calibration is consistent across every axis.
4 Strong	Approve	Active verification with local build or execution evidence, solid turn citations, calibrated preference. For REQS tasks, verified line-level patch citations with no run hygiene flaws. Approve, and state in plain terms what reaching a 5 would have taken.
3 Adequate	Request changes	Active verification with minor omissions, or basic test execution with minor slips. Send back for rework.
2 Verification defect	Request changes	Code graded from logs without local test execution or assertion verification, including quoting counts from `session.jsonl`; behavioral tags with no turn numbers; a comparison score that contradicts the per-arm ratings; mismatched follow-ups or run sequence anomalies with no explanation in `run.json` notes; schema errors (an empty items list, stray files such as `verification.md` or `.gitkeep`). Leave a 1 to 3 sentence comment, written to the fellow, stating the exact commands and actions that unlock approval.
1 Unusable	Request changes	Setup or grading text leaked into prompts or transcripts, fabricated turn numbers, a missing arm, or an empty submission.

If you reject or request changes, write down the exact transcript turn or timestamp that shows the problem.
Record your own verdict separately from any automated QC comment, especially when you are overriding it.