Project Boson
You run one real coding task twice with a terminal coding agent, once as Arm A and once as Arm B, grade both runs, and tell us which was better and why.
Important
Your ratings, rationales, behavioral issues, severities and comparisons must be your own judgement. Using any LLM or AI tool (ChatGPT, Claude, Copilot, Cursor, or any other) to write, judge or check them is prohibited. The one exception is the project assistant, and only for objective facts and tests, see Set Up the Task → The assistant. Any use of AI for a judgement call, including asking the assistant a banned question, will result in immediate removal from the project and offboarding.
What your evaluations are for
- Measure which model professional engineers prefer working with, on realistic coding tasks, inside the real product.
- Catch behavioral regressions that automated tests miss: laziness, false claims of success, over-engineering, destructive actions.
- Produce transcripts that are used to build new, high-signal evaluations.
That is why the grading asks for specific evidence, not impressions: your ratings and behavioral tags are the signal.
How this site is organized
Ways of Working
Communication
#boson-tasking: questions and blockers.#boson-announcements: updates from the team.
Workspaces
HAI: claim the task, get your task repo link and API key. Your task repo on GitHub: everything else. Your machine: where hs runs the agent in Docker.