Fresh Eyes Beat a Second Family

thirdshift is my factory for unattended coding. It takes a GitHub issue and returns a pull request. A Run builds one issue. A Spec run builds a feature that has been broken into Tickets, then reviews the finished branch as a whole in a Spec review. Weeding has the factory write its own refactoring plans while I sleep, through Architect runs. Each session runs on a coding-agent CLI, which thirdshift calls a Harness: Claude Code, Codex, and four newer ones.

Since about September 27, no human has read a factory pull request before it merged. They merge themselves. The only reviewer is the agent's own review: the session that wrote the code starts two reviewer sub-agents of the same model, then decides which of their findings to fix. The numbers made me uneasy:

• 1,152 findings declined across 199 merged PRs. Only 12 of those declines left the call to a human or a later issue, and nobody read the declines.
• 98% Strong. 209 Architecture reviews rated their own top pick Strong 98% of the time, which let them publish 202 plans. Nobody else checked a plan.
• 16 bugs reached main on thirdshift itself. Six were timing bugs: what happens when two Runs act at once, or when a step repeats.

The obvious fix is to have a different model family check the work. The intuition has support. Zheng et al. (2023) found LLM judges favor their own answers, and Panickssery, Bowman and Feng (2024) found models recognize their own text and rate it higher. If my Claude reviewers share Claude's blind spots, Codex might not. That became issue #429.

I didn't want to build a feature on an intuition. Every review I wanted to test had already happened once, and I knew which bugs it let through, because they had been found and fixed since. So I replayed those reviews.

Each session got a throwaway repository holding the history only up to the commit under review. There was no remote, no later commits, and a signed-out gh. The prompt forbade builds, tests, network access and edits. Sessions ran one at a time, 183 of them, between October 10 and 11, 2026. Every one exited before its 2.5-hour timeout, and none changed its repository. I checked HEAD and git status before and after each.

Two design choices did most of the work.

• A fresh-session control. Comparing Codex against the recorded review mixes two changes: a different family, and a reviewer who didn't watch the code being written. So most replays also ran the same model in a fresh session. If fresh Claude catches what Codex catches, the family isn't what helped.
• Bars fixed before any session ran. This is the logic of preregistration (Nosek et al., 2018): a bar set after the results are in tends to land where the results are. I settled every bar during a 33-question grilling, before the script existed.

Fig. 1 shows the four checkpoints I tested and where each sits in the factory. It's colored by result, so you can see the ending now. The rest of the post explains why each piece landed where it did.

The first replay took the five PRs behind the six timing bugs and reviewed each one as it stood when it merged. A Second opinion would have seen exactly that. Every arm ran thirdshift's current review skill, which asks each finding for a test that fails if the finding is right. Grading was blind: one packet per PR, arms behind shuffled labels, and each finding judged against the bug's real fix and a written catch criterion. A finding had to name the defect and tie it to its consequence. Getting close was a partial, and partials don't count.

The bar: the other family catches at least 2 bugs that fresh Claude misses. Toggle partials on in Fig. 2 and watch the bottom row. The bar doesn't move.

Codex, GPT-6.1 Sol at its best setting, xhigh, caught one bug, #182. Fresh Claude caught that one too, so Codex added nothing. The four newer Harnesses caught one between them, and fresh Claude had that one too.

"Codex is just the weaker model." That's a fair objection. On Artificial Analysis's index, GPT-6.1 Sol at xhigh scores 51.0 and Claude Opus 5.5 at max scores 57.6. So I ran it the other way round. Three bugs have reached main from PRs Codex wrote. Claude reviewing those PRs caught none, with two partials, and fresh Codex caught none either. A stronger model in the second seat didn't rescue the idea. Neither did a weaker one.

The timing checklist tied. A fresh agent that hadn't seen the bugs wrote it from the project's glossary and ADRs. With it, Claude caught 4, the same count as without it, but not the same 4. It found #24 and lost #102. A checklist that trades one bug for another isn't worth the extra 5 minutes of median review time.

The most useful row in Fig. 2 is the last column. The factory's original in-session reviews had raised two of these six bugs, #102 and #303. In both cases the author declined them as decisions for someone else, and nobody read the declines. Fresh Claude caught 4. I can't split that gain between "fresh session" and "today's review skill", because the replay changed both at once. Either way, it isn't a second family.

The plan check re-applies the Architecture review's own four tests to the plan it published. Is the friction real in the code? Does the deletion test hold? Is the design settled? Does it contradict no ADR? The check reads the plan, the glossary, the ADRs and the code as they stood when the plan was written. It never reads the session that wrote the plan. The bar: it stops at least 3 of 20 plans, and most of its stops are right. Fresh Claude agents graded every stop without knowing which arm made it.

Codex stopped three-quarters of Claude's plans, and two-thirds of those stops didn't hold. A check that blocks that often, and is wrong that often, would stall Weeding for nothing. Fresh Claude stopped half and was right 8 times in 10.

The catch was that the arm that passed didn't match how the factory runs. Since October 6, Weeding has written its plans on Codex, and on Codex's plans neither of the first two arms passed. Claude had checked them only at high effort, though, while fresh Claude had checked its own plans at max. So after publishing, I ran Claude at max over Codex's 20 plans.

It stopped 3, and the grader agreed with 2. That meets the bar with nothing to spare. Its two right stops were the same two plans fresh Codex had stopped, for the same reasons: each would have built code that reported an interrupted step wrongly. Claude's edge was one fewer wrong stop, 1 against Codex's 2. This is the one checkpoint where the other family came out ahead, and it did so on precision, by a single plan, with one session per cell. Both passing arms had Claude at max as the checker, whichever family wrote the plan, so I read this as being about the checker's strength more than its family. Whether to build it is open again.

The last replay took all 30 Spec branches, as they stood just before their recorded Spec review, and gave each one to a second reviewer. This bar is looser than the bug replays', and the two tests aren't the same. The bug replays asked whether a reviewer caught a specific known escape. This one asked whether it found any real behavior bug that the recorded review missed. The bar was such bugs in at least 6 of the 30 Specs, with no more than half the graded findings wrong. Fresh agents graded every finding against the code, as real, wrong or unsure.

The plan included only a Codex arm. Codex passed, but by then fresh Claude had matched or beaten Codex everywhere else, so I added 30 fresh Claude sessions to tell the family apart from the second look. Before you open Fig. 4, make a guess.

Both families pass easily. Codex found real missed bugs in 16 Specs, and only 2 of its 25 findings were wrong. Fresh Claude reached 19 Specs, but 9 of its 40 findings were wrong, mostly flagging behavior the Spec itself asked for. The two mostly found different bugs. Codex had 13 that Claude didn't raise, in 10 Specs. Claude had 23 that Codex didn't, in 18. Many are minor: rare races, storage failures, parsing edge cases. The findings come from six of my repositories, some private and some still unfixed, so the per-bug detail stays in private grading files.

Why does a second look pay here and not in a Run? My guess is that a Spec review is the first time anyone sees all the Tickets together, and the bugs between Tickets live there. But the bars differ too, and I can't separate the two explanations.

I also tested moving the Spec review itself onto Codex. The recorded Claude reviews raised behavior bugs in 23 of the 30 Specs. Codex alone raised them in 22, so swapping would give up catches rather than add them. That fails.

• Building: an opt-in Second opinion that runs after the Spec review, with the Harness, Model and Effort as settings. The open choice is the default. My lean is a fresh session on the Spec run's own Harness, since that did as well as the other family and needs no second login.
• Not building: a Second opinion inside each Run, the timing checklist, the four newer Harnesses as reviewers, or a Spec review that runs on the other family's Harness.
• Undecided: the plan check, as an opt-in with Claude at max as the checker.
• Already shipped, and probably more important: evidence-backed findings. The original reviews raised 2 of the 6 timing bugs, and the authors talked themselves out of both. Now a reviewer's finding comes with a test, and the author has to run it. A failing test obliges a fix. Finding the bug was never the whole problem. Acting on it was.

Claude graded Claude. Every grader was a Claude agent, and Claude is one of the two families being compared. Shuffled labels hide which arm wrote a finding, but Codex and Claude write differently enough that a grader could guess. I asked fresh agents to grade instead of grading myself, so this weakness is mine to own.

One session per cell. I didn't measure run-to-run variation. A tie, or a one-bug gap, is inside the noise. That includes the question that matters most for the default: whether Codex's 13 bugs that only it found are more than a second fresh Claude session would have turned up by chance.

The factory moved while I measured. thirdshift now runs everything on Codex at high, so today's plans, Spec branches and Spec reviews are Codex's. Codex had run only four Specs when I started, too few for a replay against the bar of 6. After publishing, I ran a pilot anyway: a fresh Codex session on each of those four, after its recorded Codex Spec review. It found real missed bugs in 2 of the 4. All 3 of its findings were real, and none were wrong. One was half of a notification bug, #603, that later reached main and has since been fixed. Neither reviewer caught it when I replayed the Ticket PRs it came from. Four Specs prove little, so my plan stands: switch the Second opinion on, let roughly ten Spec runs go by, and count real fixes from the reviewer reports thirdshift already keeps. That count is the test I couldn't run in replay.