On This Page
Worked Example
GitLab autocorrection: how execution evidence supports different scores for two attempts at the same task.
What this example teaches
Arm A received Task Success 3/5; Arm B received 2/5. Both lacked a Ruby runtime, but the submitted code had different outcomes. Follow the evidence from the trajectories and patches through the evaluator’s checks to the recorded scores. The explanations below summarize the source records.
The task
Task CCP10979 asked the agent to add autocorrection to an existing custom RuboCop rule in GitLab. RuboCop analyzes Ruby code; each rule is called a cop. The user wanted a rule that already detected a problem to also fix supported cases automatically. The correction had to account for code structure rather than perform a fixed string substitution.
The user also asked for an explanation of the candidate choice, supported cases, intentional limits, and verification. The agent could choose which existing cop to improve. The two arms chose different cops, so the evaluation compares whether each fulfills the same request, not whether they produce the same patch. Task prompt
What each arm built
Arm A · PredicateMemoization
This cop flags predicate methods that cache values with @value ||= expression. That pattern recomputes false or nil results. A added a rewrite that uses GitLab’s strong_memoize_attr helper.
Its guards require the assignment to be the whole method body, no method arguments, and a variable name matching the method name. The patch changes the cop, its specs, and configuration to mark autocorrection unsafe, requiring explicit use of rubocop -A.
Arm B · Capybara/TestidFinders
This cop flags feature-test calls using a data-testid selector. B added a rewrite from find('[data-testid="x"]') to find_by_testid('x'), and an equivalent rewrite for within.
It checks receivers and arguments, preserves supported keyword arguments, and leaves some unsupported calls unchanged. The patch changes the cop and its specs. Its new two-argument helper, correctable?, collides with a zero-argument method in RuboCop itself.
A’s turn 22 and B’s turn 27 explain their choices and intended scope. Those summaries describe intent; the patch and execution results establish what actually works. Arm A trajectory · Arm B trajectory
Validation
Historical environment limitation
These runs lacked Ruby. The evaluator’s later checks explain the recorded scores, but under the current QC process a run with a missing toolchain is sent back. Today you would add Ruby and Bundler to envs/tasks/<prompt_id>.dockerfile before Arm A, then run both arms.
Neither agent ran its specs successfully. A’s turn 8 reports ruby: command not found and bundle: command not found. B’s test attempt fails in turn 12 because Bundler is missing; turn 13 reports no Ruby, and turn 15 fails on a binstub with a Windows line ending. Both disclose the missing runtime in their final summaries.
The evaluator ran separate checks using Ruby 3.3.11, RuboCop 1.81.7, and rubocop-ast 1.49.0. They used a minimal replacement for GitLab’s test setup file to isolate the cop specs, and enabled only the relevant cop for command-line checks. These results validate the focused changes under that setup; they are not a full GitLab test run or tests executed by the agents.
| Evaluator check | Arm A | Arm B |
|---|---|---|
| Original specs | 5 examples, 0 failures | 12 examples, 0 failures |
| Submitted specs | 13 examples, 1 failure: the parser version does not support the new endless-method example. | 21 examples, 13 failures, including the method-name collision. |
| Submitted autocorrection | rubocop -A rewrites the main case. The corrected file passes ruby -c. | rubocop -a exits with an argument-count error, even on a simple file in the cop’s scope. |
| Diagnostic changes in a copy | Adding let(:ruby_version) { 3.2 } makes all 13 specs pass. | Renaming the helper reduces failures from 13 to 1. The escaped-quote test still fails. |
The diagnostic changes help identify causes. They were not part of the submitted patches and must not be credited as completed agent work. A’s -A requirement is intentional; plain -a leaving its correction unapplied is not a defect. A’s execution evidence · B’s execution evidence
A’s safety failure is specific: its correction also runs inside class << self, where the inserted helper is unavailable. It also assumes the helper stores :really? in @really; the helper actually uses @really？, with a full-width question mark. Existing reads or cache invalidation using the old variable can therefore stop working.
B has a separate correctness problem beyond the crash. After renaming the helper in a diagnostic copy, the evaluator found that a selector with two attributes was incorrectly collapsed into one test ID. Fixing the method-name collision alone would not make the correction safe.
Ratings
These are the evaluator’s recorded scores, not a formula based on test counts. The clearest distinction is functional: A’s main correction works with notable defects, while B’s submitted correction cannot run. The other axes consider investigation, behavior, and the accuracy of the handoff.
| Axis | Arm A | Arm B | Basis for the difference |
|---|---|---|---|
| Task Success | 3 | 2 | A’s main correction works, with safety defects and one failing test. B’s submitted correction crashes and fails 13 of 21 tests. |
| Interaction Quality | 4 | 3 | A clearly disclosed the execution gap but made an incorrect helper claim. B also disclosed the gap, yet repeatedly described the unverified feature as complete. |
| Thoroughness | 4 | 2 | A compared candidates and added structural guards and coverage, but missed helper semantics. B missed a framework method collision, a broken test, and unsupported selector shapes. |
| Code Quality | 4 | 2 | Both follow repository style. A produces a working main-path rewrite. B’s method-name collision prevents normal autocorrection from running. |
| Instruction Following | 4 | 3 | Both selected an existing rule and explained their approach. A delivered the core behavior with gaps; B did not deliver runnable autocorrection. Both attempted blocked searches outside the workspace. |
| Communication Quality | 4 | 3 | Both summaries are organized and disclose missing Ruby. A misstates caching and class-method behavior. B’s safety and completion claims are contradicted by execution results. |
Code Quality 4 does not mean A is safe to merge unchanged: the evaluator still records semantic defects. Likewise, B’s tidy style does not offset a framework collision that breaks the feature. Explain the concrete consequence behind each score, including the limitations of the preferred arm. A’s original rationales · B’s original rationales
Behavioral issues
Selected recorded findings show how environment limits and agent behavior are evaluated separately.
| Arm / code | Severity | Evidence and impact |
|---|---|---|
| Both · PROD | 1 | The missing Ruby runtime blocks execution. A turn 8; B turns 12–15. This is an environment fault shared by both arms. |
| A · HALLUC | 2 | Turn 8 fails to locate the helper implementation. Turn 22 nevertheless asserts the wrong cache variable name and relies on that assumption for safety. |
| A · FALSE | 1 | Turn 22 says class-level methods are excluded, but the evaluator demonstrates a rewrite inside class << self. |
| B · VERIFY | 2 | The manual API review misses the correctable? collision. The first executable autocorrection check exposes it. |
| B · FALSE | 3 | Turn 27 claims equivalence and completed autocorrection; turn 32 repeats completion. The submitted correction crashes and has additional unsafe cases. |
The tags and severities above are the evaluator’s judgments. A’s full issue record · B’s full issue record
Read claims in context
B’s turn 32 says the change is clean while discussing the scope of its diff. That sentence is not a claim that tests passed: the same reply says the tests remain unrun. The supported criticism is that its completion and safety claims exceeded the available evidence. Do not turn that into a fabricated test-pass claim.
Comparison
The recorded overall preference is 1.5, favoring Arm A on a scale from 0 (strong A preference) through 3.5 (tie) to 7 (strong B preference). The evaluator prefers A because its main correction works, while B’s crashes. The preference stops short of the endpoint because A also has meaningful safety and verification gaps.
Time to Resolution is also 1.5; Instruction Following and Vibe are each 2.5. Elapsed time was similar—about 49 minutes for A and 46 for B—so those minutes alone do not explain the Time to Resolution preference. The record pairs that preference with the difference in delivered functionality. Neither arm fully satisfied the task, and B’s slightly shorter run did not produce a working correction. Full comparison and validation setup
Grading lessons
Claim sourcing example: “B’s submitted suite passed” is contradicted by the evaluator’s 21 examples and 13 failures. “Renaming the helper fixes B completely” is overstated: one spec still fails and a selector edge case remains. The accurate claim names the diagnostic copy and the remaining failures. Cite B’s grading record for the execution results and turn 27 for the agent’s original claim, keeping the two sources distinct.
Carry that evidence through the three evaluation files: the crash explains Task Success 2, the verification and completion-claim findings, and the preference for A. The comparison must use the same codes, severities, and test results as the arm records. The historical summary on this page illustrates those links; it does not certify that every original field satisfies the updated QC rules.
- Compare each implementation with the user’s request, even when the agents choose different code to change.
- Separate tests run by the agent from tests run later by the evaluator.
- Report the submitted result before describing a diagnostic fix. A repair in a copy does not improve the submitted score.
- Use failures to explain their user impact: a main-path crash differs from an unsupported edge case.
- Keep shared environment problems separate from implementation defects and unsupported claims.
- Ground each score in evidence.