One Mind to Build, One Blind Mind to Break
Does splitting a coding agent into isolated "maker-checker" roles beat a single-context coding session? Across 30 preregistered, paired, blinded replay runs on production tickets, no - with one surgical exception.
- Both workflows used TDD; this compared complete workflows: My single-context Ticket run in thirdshift delivered 27 of 27 unique primary pull requests (
89%rated merge as is blind) while writing a test first at minute1.3–1.8. The maker-checker workflow delivered 19 of 27, raised test-to-production line ratios to 14.7:1 and 10.10:1, and cost $422.84 and about 49.5 active hours across the 30 pairs vs. $39.32 and 8.0 hours, summing elapsed trial time in both studies. Study 1 also differed in orchestrator and review configuration; Study 2 equalized the top-level orchestrator while retaining other workflow differences. - Equalizing the orchestrator halved cost, not failure modes: Moving the multi-agent orchestrator from Claude Code to Codex in Study 2 dropped the median cost ratio from
10.4×to4.62×, yet the blind maintainer still preferred the single-agent run 8–0–3 (and a blind colleague scored 1–0–1 on two sampled pairs), while a 17-line web server access-control rule trapped the multi-agent pipeline in an 11-hour 39-minute, 64-session, $64.22 loop with no PR opened. - The one mechanism worth keeping: Post-hoc, diff-blind behavioral execution exposed real bugs in 2 of 19 delivered pairs (Study 1 Pairs 6 and 7) that single sessions and diff reviewers missed - falling to 0 of 11 in Study 2. Instead of pre-code handoffs, I am testing deterministic
tests_on_basechecks on every ticket and a selective, runner-bounded+1Blind Behavioral Breaker fed only public signatures (public_api_delta.txt); an initial 4-pair follow-up replay cut false positives to0/4at ~$0.32API per gated task, caught an unguarded zero-date variant in Study 1 Pair 7, and missed Pair 6's deeper inline-signature bug within139test lines.
Prefer the scientific paper format?
This essay synthesizes two preregistered, blinded experimental papers written in full academic format with methods, per-task tables, statistical tests, and typeset PDFs:
Research Paper 1 · Adoption Gate (10 Issues)
Ten Replayed Issues: Maker-Checker TDD vs. a Single-Agent Ticket Run
Preregistered 10-pair adoption test (Claude Code orchestrator vs. thirdshift Ticket run): experimental design, blind human and LLM grading tables, session telemetry, and Pair 6 cross-test analysis.
Research Paper 2 · 20-Pair Census
Maker-Checker TDD vs. a Single-Agent Ticket Run: Seventeen Tasks & Three Repeats
Preregistered 20-pair descriptive study equalizing the Codex orchestrator: hypotheses H1–H6, 8-model position and family bias, T3 11.7-hour loop trace, and five governing CS mechanisms.
Separating coding agents into isolated planner, test-writer, implementer, and multi-round reviewer roles degraded delivery, multiplied cost by 4.6× to 10.4× across 30 paired replay runs, and produced half as many merge-ready pull requests as one continuous coding session - yet executing one independently written behavioral test suite after the fact caught real defects that a single session and its reviewers missed.
I ran these experiments because I carry the merge gate. At Educational Travel Adventures, most bug fixes and small features on Tourbot - our million-line PHP 8 and MySQL 8 operations platform - start inside thirdshift, an open-source Rust orchestrator I built to run coding agents cleanly. One Codex session reads an issue, writes a failing test first, builds the change, spawns parallel spec, standards, and security sub-agents, and opens a pull request. That ships fast, but it breaks a rule from Cleanroom software engineering (Mills, Dyer, & Linger, 1987): nobody should grade their own homework. If one context misreads a ticket, both its code and its test suite carry the blind spot. Before migrating our upcoming feature builds onto an isolated multi-agent Maker-Checker TDD pipeline, I wanted to measure what role separation actually buys - and what it quietly breaks.
Each section below starts folded so the questions read as the argument in brief. Click any heading to open it, search natively with Find in Page, or .
If you evaluate a heavy workflow on hand-picked tickets or eyeball unblinded PRs, you only measure what you expected to see. Following Nosek et al. (2018) on preregistration, I built a deterministic replay harness and ran two back-to-back studies covering 27 unique historical Tourbot issues plus 3 seeded repeats (30 paired runs, leaving aside one shakedown run per study):
- Mechanical ticket selection: In Study 1 (10 issues, Oct 30 – Nov 1, 2026), we screened 156 closed issues down to 24 small-to-medium fixes (2–8 files, no schema migrations) and took the first 10 with explicit acceptance criteria. In Study 2 (17 new tasks + 3 repeats = 20 pairs, Nov 3–5, 2026), a committed selection script took every eligible closed issue (+13 to +330 LOC) not in Study 1, paired with a frozen numbered acceptance checklist per task.
- Identical starting trees and isolated machines: Both arms started from the parent commit immediately before the real fix merged, with post-triage comments stripped, running in parallel on cloned 8-vCPU cloud droplets with scoped tokens reaching only separate replay repositories. After completion, an automated leak scanner verified zero transcripts saw the shipped PR or commit hash.
- Blind formatting, execution evidence, and dual-order grading: A packet builder stripped branch names, commit messages, and agent scratch files, ran
php-cs-fixer(@PSR12) identically acrossbase,X, andY, randomizedX/Ylabels, transplanted each PR's tests onto the other PR's code (cross_tests) and onto the unpatched parent (base_testsin Study 2), and invoked blinded LLM graders in both presentation orders ([X, Y]and[Y, X]) to neutralize position bias (Zheng et al., 2023; Shi et al., 2025). I graded every delivered pair blind before opening any model verdict or arm mapping; in Study 2, a two-stage form locked my unaided merge calls before I opened any review helper, and a colleague independently graded two sampled pairs blind (Pairs 10 and 12).
Study 1 tested an adoption gate under the rules as packaged on October 29: a Claude Code orchestrator managing separate fresh-context Test Plan, RED Test, GREEN Code, and Refactor makers, each checked by another provider before advancing, plus up to five rounds of cross-family code review. That run rejected wholesale adoption on every threshold: 2 wins in 10 (needed 8), 10.4× median cost, 6.4× median time (cap 2.0×), 3 correctness or delivery failures vs. 2, and 1 of 8 pull requests mergeable as is blind vs. 7 of 8 for the single-agent Baseline.
Fair criticism remained after Study 1: Claude's API-rate pricing is higher than Codex's cached tokens, and 64 commits landed on the Candidate rules between October 29 and November 3. So in Study 2 we ran both arms under the exact same top-level Codex gpt-6.1-sol (high) orchestrator on 17 new tasks plus 3 repeats, adding base_tests and a 6-family non-arm voting panel plus 2 exact-model diagnostic graders. Moving the orchestrator to Codex halved the relative cost and time ratios (10.4× → 4.62× cost; 6.4× → 3.28× time), yet widened the delivery gap (17/17 vs. 11/17) and left blind maintainer preference essentially unchanged (8–0–3 Baseline across delivered first pairs; 10/11 vs. 2/11 mergeable as is).
The single-context Builder delivered 27 of 27 primary PRs; isolated maker-checker roles delivered 19 of 27. The proposed selective hybrid keeps one Builder and adds a bounded, implementation-blind check. Read the workflow and information-barrier contract.
public_api_delta.txt barrier keeps a selective Blind Breaker blind to implementation bodies and tests without guessing internal helper names. A static text summary remains if initialization fails.| Measure (Baseline vs. Candidate) | Study 1 (10 pairs) | Study 2 (17 primary + 3 rep.) |
|---|---|---|
Primary PRs delivered (B vs. C) |
10/10 vs. 8/10 | 17/17 vs. 11/17 |
| Median API cost per task (paired ratio) | $1.35 vs. $14.06 (10.4×) | $1.27 vs. $5.46 (4.62×) |
| Total API-priced spend across study | $13.39 vs. $223.71 | $25.93 vs. $199.13 (20p) |
| Median active minutes (paired ratio) | 10.8m vs. 62.0m (6.4×) | 18.9m vs. 55.1m (3.28×) |
| Median minutes to first PHP edit | 1.3m vs. 24.4m (21.4×) | 1.8m vs. 11.4m (6.11×) |
| Test lines per production line (delivered) | 3.7 vs. 14.7 | 2.64 vs. 10.10 (med.) |
Blind author merge as is (delivered) |
7/8 (88%) vs. 1/8 (13%) | 10/11 (91%) vs. 2/11 (18%) |
Blind author wins (B–C–tie, delivered) |
6–0–2 (cat. 7–0–1) | 8–0–3 (14–0–3 all 17) |
Colleague blind sample (2 pairs: T10, T12) |
- | 1–0–1 Baseline |
Candidate cross-test defects found in B |
2 / 8 (I6, I7) |
0 / 11 (1 B catch on T17) |
If an expensive workflow spent its extra compute writing deeper application code, the bill might buy reliability. Across both studies, the phase logs show the opposite: both arms spent nearly the exact same number of active minutes implementing the PHP fix itself (89m Baseline vs. 85m Candidate across 10 pairs in Study 1; 193.5m / $14.53 vs. 199.1m / $13.97 across 20 pairs in Study 2). Three-quarters of the Candidate's money - 74% in Study 1 and 76.0% in Study 2 - never touched shipping code at all.
Across 30 pairs, the rounded Candidate phase rows total $420.39 versus Baseline's $39.32. Orchestration, planning, and RED verification consumed $314.27 (74.8%) of that Candidate spend. The published tracked totals sum to $420.37; the two-cent difference is rounding. Read the per-study spend breakdown.
$222.67 (Study 1 sessions) + $197.70 (Study 2 tracked phases) = $420.37 ($422.84 including unattributed command overhead), vs. Baseline $39.32. The combined bars sum the rounded source phase rows to $420.39, a $0.02 rounding difference; percentages use that row sum. Separately, summing the rounded elapsed trial rows in Study 1 Table 12 and Study 2 Table 5 gives about 2,967.4 minutes (49.5 hours) Candidate vs. 482.6 minutes (8.0 hours) Baseline; these are trial totals, not sums of overlapping session times. Actual make + check tool medians barely moved between arms in Study 2 (94.0/65.0 vs. 88.5/59.0); the Candidate's median 237.5 orchestration calls drove the gap.Two classic software-engineering laws explain why role separation multiplied effort instead of quality:
- Context fragmentation vs. conceptual integrity (Brooks, 1975; Yan, 2025; Kim et al., 2025): In The Mythical Man-Month, Fred Brooks argues that a clean fix depends on keeping one coherent mental model rather than splitting coupled decisions across actors whose handoffs grow quadratically. Walden Yan's context-sharing rule updates Brooks for coding agents: exploring a legacy PHP repository builds a dense belief state in context, and every fixture line encodes implicit assumptions. Starting
11to72fresh contexts threw away that exploration trace every handoff, multiplying file reads by5.2×(Study 1:2,505vs.481) and4.0×(Study 2:2,934vs.729) and causing isolated test writers to invent internal helper names that clashed with the Baseline (T5,T10). Across 260 configurations on SWE-bench Verified, Kim et al. (2025) measured the same penalty: every multi-agent architecture trailed a single agent by2.1to14.9percentage points. - Cleanroom test separation vs. Goodhart's Law (Mills et al., 1987; Goodhart, 1975; Hora & Robbes, 2026; Chen et al., 2026): Separating test authors from coders works in DO-178C and IBM Cleanroom aerospace projects only because a complete formal specification fixes every signature and transition before a line of code exists. On a 15-line natural-language ticket over a legacy web app, an isolated RED writer cannot yet be scored on catching implementation bugs - its survival depends on pleasing adversarial verifiers before any implementation exists. As Goodhart's Law predicts, passing the verifier checklist became the target: writers generated defensive mocks, PHP source-string regexes, and bloated harnesses (
14.7:1test-to-code ratio in Study 1;10.10:1in Study 2). On a two-linesession_write_close()fix in Study 1 Pair 9, reading the tests felt like auditing a mortgage contract to buy a pack of gum.
Study 2 Task T3 asked for a web server access-control rule blocking HTTP requests to dot-paths, repository metadata, dependency manifests, and log or SQL dumps - 17 lines of configuration. My single-agent Ticket run shipped a clean patch and regression test in 15.7 minutes ($1.25, 4 sessions). The Candidate actually finished the working config fix and 543 passing HTTP assertions in 1 hour 44 minutes ($8.10). Then four natural-language rules locked into a non-terminating loop for the next ten hours.
T3 passed 543 HTTP assertions by 1h 44m, then entered repeated fixture-helper reviews until a 699.4-minute timeout: 64 sessions, $64.22, and no PR. Read the trace stages and stopping rules.
4 sessions, 15.7 minutes, $1.25, green CI, PR shipped.Every failure in the multi-agent traces maps cleanly onto two architectural results in the agent literature:
- Deterministic code workflows vs. prose-interpreted loops (Schluntz & Zhang, 2024; MAST, Cemri et al., 2025): Schluntz and Zhang draw the line between code-driven workflows (
thirdshift's fixed Rust state machine: build, three parallel reviewers, bounded repair) and prompt-driven agents interpreting Markdown rules at runtime. In UC Berkeley's MAST taxonomy of 1,600+ multi-agent traces, failures stem predominantly from orchestration rules rather than weak base models: halting unattended runs when a planning question lacks a human (FM-3.1Premature Termination:I2,I10,T1,T8,T13,T16), brushing past a maker's low-risk assessment (FM-2.5Ignored Other Agent's Input), diagnosing churn while authorizing another cycle (FM-2.6Reasoning–Action Mismatch), and looping wherever "changed evidence" exists (FM-1.5Unaware of Termination Conditions). - Appetite, scope hammering, and circuit breakers (Singer, 2019): Ryan Singer's Shape Up starts from fixed appetite - how much time a change is worth - scope-hammers non-essential polish to follow-up issues, compares improvements against the live baseline rather than an imaginary ideal, and trips a hard circuit breaker before rabbit holes eat the schedule. Natural-language review policies that forbid follow-up deferrals and replace user pauses with unbounded meta-review do the opposite: fixed scope, unbounded time.
If that were the whole story, you would bin every piece of the multi-agent workflow and never look back. But Study 1 Pair 6 (I6, replacing stale signatures in queued bid emails without touching coordinator notes) refuses to let me ignore the other side. Working in one context, my Baseline agent pictured a signature as the trailing block at the bottom of the email, wrote a broad name-line regex and happy-path tests, passed its own TDD suite, passed its parallel spec, standards, and security sub-agent reviews, and passed my blind reading as merge after small fixes.
When the evaluation harness transplanted the Candidate's independently authored tests onto the Baseline's code, they failed immediately and proved three real bugs: the Baseline deleted coordinator text written beside a signature, rewrote a historical signature copy stored inside notes, and left stale contacts in legacy templates. In Pair 7 (I7), independent tests caught a daylight-saving spring-forward failure too.
Three results explain both the Pair 6 catch and why Study 2 changed how I would deploy it:
- Intrinsic self-correction blindness vs. execution oracles (Huang et al., 2024; Kamoi et al., 2024; Knight & Leveson, 1986; Konstantinou, Tambon, & Papadakis, 2026): Language models cannot reliably spot their own reasoning mistakes without an external ground-truth failure signal, and separate reviewers shown the implementer's diff anchor on the same framing (correlated misses). Independent tests written without seeing the implementation catch nearly double the faults (
25%vs.14%; Konstantinou et al., 2026) - but you get that lift from post-hoc execution of a diff-blind test suite, not from 14 rounds of pre-code planning (Pair 6 cost 72 sessions and $74.14!). - Study 2 answered my open question cleanly (H1 = 0 in Study 2; 2 of 19 overall): Across all 11 newly delivered primary pairs (13 with repeats) in Study 2, Candidate cross-tests found zero confirmed defects in Baseline PRs (the two failures, on
T5andT10, merely asserted private Candidate helper structure), while a Baseline unit test caught one real Candidate regression (filter_var(..., FILTER_VALIDATE_INT)rejecting valid zero-padded IDs like'0010'onT17). Combined across both studies, independent Candidate tests caught Baseline bugs on 2 of 19 delivered pairs (10.5%) - and Pair 6 needed 1,736 test lines to get there. - Human attention economics vs. LLM length bias (Sadowski et al., 2018; Pombal et al., 2026): Make a prediction in Fig. 4 before revealing how blind human reviewers and eight model graders traded off test size and quality.
Across the 14 non-tied primary pairs graded blind by the human maintainer in Study 1 (6) and Study 2 (8), plus a colleague's 2 sampled blind reviews in Study 2, how many times did the blind human pick the pull request with the larger test diff?
sonnet, astra, gemini, grok, muse, kimi) plus 2 same-model diagnostics (opus, sol), each evaluated in both orders. OpenAI models tolerated large test diffs far more often (4/9 for astra, 3/9 for sol); dropping astra shifts three primary ties to Baseline wins (8–1–2) plus the repeat T15-r2, and excluding both OpenAI and Anthropic drops Candidate wins to 0 (5–0–6).Reconciling the November research data leaves two clean tiers in thirdshift: cheap deterministic guardrails that run on every ticket at essentially zero marginal cost, plus a selective, runner-bounded +1 Blind Behavioral Breaker (or feature-boundary review) where behavioral mutations warrant it.
- On every ticket (~$0, seconds): Keep the single continuous-context Builder (
Explore → write TDD test → implement → green CI), seton_ambiguity = "assume_and_document"so minor questions become stated PR assumptions rather than unattended halts (0/27halts instead of6/27), run deterministictests_on_baseagainst the unpatched parent commit so every regression test proves it fails without the fix, instruct code reviewers to flag source-string regex assertions and test-helper spirals, and enforce loop ceilings in Rust runner code. - Selective
+1Blind Behavioral Breaker (and Spec-level second look): Paying for an extra isolated test session on dead-script deletions (T6,T12) returned zero defects in Study 2. Instead, reserve a single bounded cross-family breaker (≤ 180s,≤ 25tool calls,≤ $1.50,≤ 150LOC /5test methods) for stateful data-mutation tickets (like Study 1 Pair 6 and Pair 7) or the whole-featureSpecboundary where Fresh Eyes found real missed bugs in 16–19 of 30 features. - Solve helper mismatches with
public_api_delta.txt: In Study 2, Candidate cross-tests falsely failed on Baseline PRs twice (T5,T10) because an isolated test writer guessed private helper names. Generating a stripped signature header - showing only added or changed public class/method signatures with every method body replaced by{ /* hidden */ }and alltests/diffs omitted - fixes helper-name mismatches while keeping the breaker blind to implementation flow.
Selective 4+1 runner contract (reconciled post-Study 2):
# thirdshift selective 4+1 hybrid configuration
[limits]
# 1 builder + 3 diff reviewers + 1 selective breaker + max 1 repair
max_total_sessions = 6
max_wall_clock_minutes = 30 # hard cap (vs. 699m in prose loops)
max_api_cost_usd = 4.00
on_ambiguity = "assume_and_document" # avoids stopped-to-ask halts
[steps.build]
# one mind: explore -> red test -> implement -> tests_on_base
context = "continuous"
[steps.extract_public_signatures]
run = "thirdshift extract-signatures --base HEAD~1 --head HEAD"
out = ".thirdshift/public_api_delta.txt"
[steps.verify_parallel.blind_breaker]
# skip trivial deletions and config rules (Study 2 H1 = 0)
enabled_for = ["stateful_mutation", "spec_boundary"]
provider = "claude" # cross-family executable probes
input = ["issue", "base_repo", "public_api_delta"] # no diff bodies/tests
max_duration_seconds = 180
max_tool_calls = 25
max_test_lines = 150
max_test_methods = 5
retain_test_only_if = "fails_on_head_or_increases_branch_coverage"
[steps.repair_if_needed]
type = "resume_builder" # keeps warm KV context intact
max_rounds = 1 # never spawns RED->GREEN helper loops
Follow-up replay (November 18): testing the bounded Blind Breaker on four saved pairs
Before leaving the configuration above as an untested sketch, I replayed four saved tasks from Studies 1 and 2 on November 18 using those exact bounds (claude-opus-5-5 at medium effort, ≤ 180s, ≤ 25 tool calls, ≤ 150 test lines, ≤ 5 test methods, public_api_delta.txt with function bodies and tests/ stripped; full log in follow-up-replay.md): Study 1 I6 and I7 (the two Baseline defects exposed by 1,736-line and 1,808-line Candidate suites in Study 1) plus Study 2 T5 and T10 (the two tasks where Study 2 Candidate cross-tests failed solely on mismatched internal helper names). I also forced T5 to run despite its zero-signature gate so we could verify false-positive rate on a non-PHP configuration task.
| Task (Study) | Public API & Gate | Size / Time / API Cost | Base Check | Target PR Outcome (Baseline / Candidate) |
|---|---|---|---|---|
Study 1 I6 (signature repointing) |
5 sigs / Run | 139 LOC / 61.7s / $0.35 | 5/5 fail (expected) | Baseline 5/5 pass (29 assertions); missed deeper inline-note case |
Study 1 I7 (zero call-date reset) |
2 sigs / Run | 106 LOC / 54.1s / $0.27 | 5/5 fail (expected) | Baseline 1 error / 4 pass (threw on '0000-00-00 00:00'; 40 assertions) |
Study 2 T5 (container webroot) |
0 PHP sigs / Skip* | 123 LOC / 56.3s / $0.32* | 2/5 fail (expected) | 5/5 pass on both arms (39 / 41 assertions; 0 false positives) |
Study 2 T10 (deck override apply) |
1 sig / Run | 128 LOC / 65.5s / $0.34 | 5/5 fail (expected) | 5/5 pass on both arms (46 / 46 assertions; 0 false positives) |
*Note on T5 and cross-arm runs: With selective gating turned on, T5 extracts zero PHP signatures and skips the breaker for $0.00. Because public_api_delta.txt was extracted from Baseline's pull request, running the I6 and I7 probes against Candidate rightly stopped at the missing-function signature guard; in production the breaker inspects the builder branch's own extracted signatures.
Two crisp lessons fell straight out of those four runs. First, public_api_delta.txt eliminated both Study 2 false positives (T5 and T10 went from 2/2 spurious cross-test failures to 0/2) while costing $0.95 API total across the three gated runs ($0.27–$0.35 each in 54–66 seconds; CLI-reported equivalent $1.26). Second, the 150-line budget showed an honest depth trade-off: on I7, 106 lines of black-box boundary inputs surfaced an unhandled zero-date sentinel missing seconds ('0000-00-00 00:00', throwing an uncaught InvalidArgumentException alongside 40 passing assertions), whereas on I6, 139 lines tested clean role handoffs, HTML whitespace reflow, and a standalone coordinator notes block - passing all 29 assertions on Baseline without discovering the trickier defect Study 1 Candidate exposed after 14 plan rounds and 1,736 fixture lines (coordinator notes typed inside the trailing signature block or a pasted signature inside Notes). Short bounded probes give you a cheap, clean interface oracle; deep adversarial corpus testing belongs selectively on high-risk text parsers and full feature branches.
When I'd choose differently: If I were building a cryptographic protocol, a financial settlement engine, or an avionics controller with a formal specification written down to exact invariants before any code exists - and where a $75 verification bill per change is trivial next to a seven-figure failure - Cleanroom role separation makes sense. On 2–8 file tickets in an evolving web application shaped from 15-line issues, separating test authors from builders destroys context without buying safety.
What I still don't know:
- What happens when we scale the November 18 bounded Blind Behavioral Breaker across the remaining delivered Baseline pairs? The initial 4-pair replay answered the first half: a 139-line pass stayed cheap (
$0.35API), kept false positives at zero, and missed Pair 6's subtler inline-signature corruption, while a 106-line pass caught an unguarded'0000-00-00 00:00'edge case in Pair 7. Whether a slightly higher adversarial budget on risky diffs or feature-level runs recovers Pair 6 without ballooning back toward bloat stays open. - Where should the boundary sit between ticket-level probes and feature-level probes? Study 2 found
0/11Baseline bugs on small tickets, whereas Fresh Eyes found missed bugs in more than half of finished multi-ticket Spec branches. Running both checks cleanly on upcoming feature builds will tell. - How do we run fast mutation testing (
Infection) on legacy procedural PHP files outsidesrc/? Seven of ten pairs in Study 1 and many procedural tasks in Study 2 generated zero mutants, leaving an instrumentation blind spot.
| Dimension | Single-Agent Ticket Run | Multi-Agent Maker-Checker | Selective 4+1 Hybrid (target) |
|---|---|---|---|
| Cost efficiency (API proxy) | 9 / 10 ($1.27–$1.35) | 2 / 10 ($5.46–$14.06) | 8 / 10 (+~$0.30–$0.50 gated) |
| Wall-clock latency & delivery | 9 / 10 (27/27 PRs) | 3 / 10 (19/27; 11.7h tail) | 9 / 10 (hard 30m cap) |
| Diff cleanliness & merge readiness | 9 / 10 (89% as is blind) | 2 / 10 (16% as is; 10–15:1) | 9 / 10 (prune redundant probes) |
| Subtle blind-spot fault catching | 6 / 10 (missed I6, I7) | 7 / 10 (caught I6, I7; missed I8, T17) | 8 / 10 (targeted blind oracle) |
| Operational predictability (no loops) | 9 / 10 (fixed state machine) | 2 / 10 (FM-1.5 / FM-2.6 loops) | 9 / 10 (max_rounds = 1 in code) |
Equalizing the orchestrator lowered token prices, but orchestration still took 56.1% of Candidate spend and green coding still spent $13.97 vs. $14.53.
All 543 HTTP policy tests passed at 1h 44m ($8.10). The remaining 9h 55m spent 1,101 lines and a 32-row state table debating a test helper's cleanup method.
That 2/19 rate is why the +1 Blind Breaker should be gated selectively onto stateful data-mutation work or whole-feature reviews rather than trivial config or file-deletion tasks.
public_api_delta.txt stop false-positive cross-test failures like T5 and T10 without breaking independence?Giving public signatures prevents guessing private helper names while hiding method bodies prevents anchoring bias (Huang et al., 2024; Konstantinou, Tambon, & Papadakis, 2026).
In Study 2, humans saw cross-tests and base-tests inside the blind packet. Even with identical evidence, the blind author (8-0-3) and colleague (1-0-1) never once picked the larger diff.
Those four rules eliminate 6/6 planning halts and runaway test-helper loops while proving regression tests fail on the unpatched commit.
0 of 6 answered.
Companion studies & software
- Study 1 (10-issue adoption test): How a Maker-Checker, Test-Driven Process and a Single-Agent Ticket Run Differ (Ten Replayed Tourbot Issues) · PDF (26 pp.)
- Study 2 (17 tasks + 3 repeats, Codex orchestrator): How a Maker-Checker, Test-Driven Process and a Single-Agent Ticket Run Differ (Seventeen Tasks & 8-Model Panel) · PDF (33 pp.)
- Follow-up replay & reconciliation record: Four-pair bounded Blind Breaker replay and Issue #69 reconciliation (Markdown)
- Prior review study: Fresh Eyes Beat a Second Family (183 replayed code reviews showing why post-feature review finds real missed bugs).
- Orchestrator under test: thirdshift (github.com/JacobStephens2/thirdshift).
Primary research & engineering literature
- Frederick P. Brooks, Jr., The Mythical Man-Month (1975) & "No Silver Bullet" (1986): conceptual integrity and quadratic coordination overhead.
- Walden Yan, "Don't Build Multi-Agents: Principles of Context Engineering" (Cognition, June 2025): sharing full context traces vs. colliding implicit decisions.
- Yubin Kim, Ken Gu, Chanwoo Park, et al., "Towards a Science of Scaling Agent Systems" (December 2025): 260 multi-agent configurations on SWE-bench Verified. arxiv.org/abs/2512.08296
- Harlan D. Mills, Michael Dyer, & Richard C. Linger, "Cleanroom Software Engineering" (IEEE Software, 1987): specification-complete separation of coding and verification.
- Charles Goodhart (1975) & Marilyn Strathern (1997): metric targeting - when verifier approval becomes the optimization target.
- Mert Cemri, Matei Zaharia, Ion Stoica, et al., "Why Do Multi-Agent LLM Systems Fail?" (MAST taxonomy, 2025). arxiv.org/abs/2503.13657
- Erik Schluntz & Barry Zhang, "Building Effective Agents" (Anthropic, December 2024): deterministic workflows vs. open-ended agent loops.
- Jie Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR, 2024); Ryo Kamoi et al., "When Can LLMs Actually Correct Their Own Mistakes?" (TACL, 2024); John C. Knight & Nancy G. Leveson (IEEE TSE, 1986); Michael Konstantinou, Florian Tambon, & Mike Papadakis, "On the risk of coding before testing: An empirical study on LLM-based test generation workflow" (
25%vs.14%, July 2026, arxiv.org/abs/2607.05139); Zuodong Xiang, Yike Zhang, YueMing Zhang, & Hailu Xu, "Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?" (Agentic SE @ KDD '26, arxiv.org/abs/2607.21656). - Andre Hora & Romain Robbes (MSR, 2026, arxiv.org/abs/2602.00409); Zhi Chen et al. (2026, arxiv.org/abs/2602.07900); Caitlin Sadowski et al., "Modern Code Review: A Case Study at Google" (ICSE, 2018); José Pombal, Ricardo Rei, & André F. T. Martins, "Self-Preference Bias in Rubric-Based Evaluation of Large Language Models" (April 2026, arxiv.org/abs/2604.06996); Bjarni Haukur Bjarnason, André Silva, & Martin Monperrus, "On Randomness in Agentic Evals" (February 2026, arxiv.org/abs/2602.07150).
- Brian A. Nosek et al., "The preregistration revolution" (PNAS, 2018); Ryan Singer, Shape Up (Basecamp, 2019).
For AI agents: agents.md contains the workflow contract, verified numbers, operational checklist, and self-test with answers.