Fresh Eyes Beat a Second Family: A Preregistered Replay Study of Cross-Family AI Code Review in an Unattended Coding Pipeline

Coding agents now write, review and merge code with nobody watching. In thirdshift [6], the system studied here, no human has read a pull request before it merged since late September 2026. Each pull request merges itself once the agent's own review is done. That review is performed by two reviewer sub-agents of the same model that wrote the code, after which the authoring session decides which findings to fix. Three facts from the pipeline's records motivated this study. First, authors declined 1,152 review findings across 199 merged pull requests, and only 12 of those declines deferred the decision to a human or a later issue. Second, 209 architecture reviews rated their own preferred plan “Strong” 98% of the time, publishing 202 plans that nobody else checked. Third, 16 bugs had reached the main branch, 6 of them timing bugs: faults that appear only when two Runs act at once or a step repeats.

A natural remedy is to have a different model family review the work. The idea has empirical support. LLM judges prefer their own answers [1], and models recognize and favor their own generations [2]. If Claude reviewers share Claude's blind spots, a Codex reviewer might not. Cross-family review is attractive for a second reason: it is easy to add to a pipeline that already supports several coding-agent CLIs.

The comparison usually made, though, confounds two changes. A reviewer from another family is also a reviewer that did not watch the code being written. We therefore ask whether a second family helps beyond what a fresh session of the same model provides, and where in the pipeline any such help appears.

Our contributions are:

1. A replay method for evaluating review policies on historical work whose escaped defects are known, with repository isolation and bars fixed before data collection (§4).
2. Results at four checkpoints of an unattended pipeline, each with a fresh same-model control, across seven reviewer configurations and 183 sessions, plus 24 follow-up sessions (§5).
3. Evidence that a fresh same-model session matched or exceeded the other family's catches at every checkpoint, and that the one checkpoint where a second look clearly paid, after a feature-level review, paid for either family (§6).

2.1 The system under study

thirdshift turns GitHub issues into pull requests. We use its vocabulary throughout. A Run builds one issue into one pull request: an implement session writes the code, starts two same-model reviewer sub-agents, and then decides which of their findings to act on. A Spec run builds a feature that has been split into Tickets. Each Ticket gets a Run, and a Spec review then reviews the finished feature branch as a whole. In Weeding, the system writes its own refactoring plans through Architect runs, publishes them, and later builds them. A Harness is the coding-agent CLI a session runs on. During the study the system supported Claude Code, Codex and four newer Harnesses. Weeding has written its plans on Codex since 6 October 2026, and by the end of the study every production session ran on Codex at high effort, so current plans, Spec branches and Spec reviews are Codex's work.

2.2 Self-preference in model judges

Zheng et al. [1] reported self-enhancement bias in LLM-as-a-judge settings: judges rated answers from their own model more favorably. Panickssery, Bowman and Feng [2] showed that models can recognize their own generations and that self-recognition correlates with self-preference. Both results motivate cross-family review. Neither tests whether a reviewer of another family finds more defects in code than a fresh instance of the same model. That is the question here.

2.3 Preregistration

Bars chosen after results are seen tend to land where the results are. Following the logic of preregistration [3], every pass/fail bar in this study was fixed during a structured design interview of 33 recorded questions before the replay script existed. The record is kept in the project's issue tracker and design log [5, 7]. Deviations are reported in §4.6.

We test four places a second reviewer could be added (Figure 1). Table 1 states each with its preregistered bar.

• RQ1 (in a Run): Does a reviewer from another family catch escaped bugs that a fresh same-model reviewer misses?
• RQ2 (plan check): Can an independent check of published refactoring plans stop bad plans without stalling good ones?
• RQ3 (after the Spec review): Does a second reviewer, run after the recorded feature-level review, find real behavior bugs that review missed?
• RQ4 (Spec review on the other Harness): Would replacing the feature-level review with the other family find as much as the original?

4.1 Replay design and isolation

Each review we evaluate already happened once, and the bugs it let through have since been found and fixed. Each replay session received a throwaway repository containing history only up to the reviewed commit: the branch under review plus a base branch for comparison, no remote, and no later commits. The GitHub CLI was signed out with an empty configuration directory. The prompt forbade builds, tests, network access and any change to the repository. We recorded HEAD and git status before and after every session, so that any session that changed its repository could be discarded. None did. The 183 main sessions ran one at a time between 2026-10-10 04:23 and 2026-10-11 20:12 UTC, and every one exited before a 2.5-hour timeout. An automated scan for GitHub CLI use flagged two sessions. Both hits were quoted search patterns, not calls.

4.2 Subjects

RQ1 used the six timing bugs that reached main from Claude-written PRs and the three bugs that reached main from Codex-written PRs, all on thirdshift itself. RQ2 used plans published by Architect runs. RQ3 and RQ4 used all 30 Spec branches with a recorded Spec review, drawn from six repositories, some of them private.

4.3 Arms

Claude ran at maximum effort where it was the baseline Codex had to beat, and at high effort where it was the second reviewer being tried. Codex ran at xhigh throughout. Each newer Harness ran at its best measured setting. On the Artificial Analysis Intelligence Index [4], GPT-6.1 Sol at xhigh scores 51.0 and Claude Opus 5.5 at max scores 57.6. Codex arms were instructed to start their sub-agents fresh, as the production system now does.

4.4 Prompts

The bug replays used the system's current review skill in every arm. That skill requires each correctness finding to carry a test that fails if the finding is right, and asks reviewers to list the files they read. The Spec replays pinned the older skill version that the recorded Spec reviews actually ran. The plan check re-applied the architecture review's own four tests to the published plan: the friction is real in the code, the deepening passes a deletion test, the design can be settled from local evidence, and it contradicts no recorded decision. It read the plan, the domain glossary, the decision records and the code as they stood when the plan was written, never the session that wrote it. A fresh agent that had not seen the bugs wrote the timing checklist from the glossary and decision records alone.

4.5 Grading

Bugs. Each bug had a written catch criterion: a finding had to name the defect and tie it to its consequence. Grades were caught (C), partial (P, not counted) and missed (M). Each PR's reports were graded as one packet, with arm labels shuffled and run paths stripped, against the bug's known fix. Borderline grades were rechecked by hand. Plans. Fresh agents graded every stop blind to the arm, verifying the stop's blocking reasons against the code at the plan's base commit. A stop counts as right if the grader would also stop the plan. Spec findings. Sorting agents listed each arm's behavior-bug findings and marked whether the recorded review had raised each. Fresh agents graded every missed finding against the code as real, wrong or unsure. A final pair of agents checked each arm's real findings against the other arm's full reports. All graders were Claude agents (§7).

4.6 Deviations from the registered plan

1. The plan had the author grading plans and Spec findings. At the author's request, fresh agents graded them instead.
2. The RQ3 plan had only a Codex arm. After Codex passed, and after fresh Claude had matched or beaten Codex everywhere else, 30 fresh Claude sessions were added over the same branches to separate family from freshness. Sessions, prompt and grading match the Codex arm's.
3. Two follow-ups were run after the first write-up (§5.5): Claude at max on Codex's plans, and a pilot of fresh Codex on Codex's own Specs. Both reuse the registered bars and grading prompts.

5.1 RQ1: a second opinion inside a Run

Codex caught one bug, #182, which fresh Claude also caught, so Codex added nothing beyond the fresh control (bar: 2). The four newer Harnesses caught one bug between them, also #182 (Appendix A). Counting partials changes no “beyond fresh” value, because every partial falls on a bug fresh Claude caught. RQ1 fails in the forward direction.

A natural objection is that Codex is the weaker model. We therefore ran the reverse: Claude at high reviewing the four Codex-written PRs behind three escaped bugs (Table 4). Claude caught none, with two partials. Fresh Codex also caught none, and so had the system's own reviews. A stronger model in the second seat did not rescue the idea. RQ1 fails in the reverse direction.

The timing checklist tied the fresh arm at four catches, but not the same four: it gained #24 and lost #102, and raised median review time from 20.1 to 25.4 minutes. Under the registered rule it is not adopted. The drop rule does not apply either, since two Claude arms caught more than one bug.

5.2 RQ2: the plan check

Twenty-two plans were stopped by at least one of the first four arms: 17 of Claude's and 5 of Codex's. Both forward arms stopped 8 of the same Claude plans; the grader upheld both stops on 5, only Claude's on 2, and neither on 1. Codex stopped three-quarters of Claude's plans, and two-thirds of those stops did not hold, a rate that would stall Weeding without benefit. Of the first four arms only fresh Claude checking Claude's plans passed. That configuration no longer matches production, because Weeding has written its plans on Codex since 6 October. The follow-up in §5.5 tests Claude at max on those plans.

5.3 RQ3: a second opinion after the Spec review

Both families pass, each with well under half of findings wrong. Most of fresh Claude's nine wrong findings flagged behavior that the Spec, or the branch's own README, asked for; Codex's two were of the same kind. Many real findings are minor: rare races, storage failures and parsing edge cases. Table 7 shows the arms' overlap. A partial match counts as “not raised”, as in bug grading, which is why the two “raised” counts differ.

A second look after the feature-level review therefore finds real bugs whichever family takes it. The same-family fresh session reached more Specs (19 vs. 16) and found more bugs. Codex was more precise (2 of 25 wrong vs. 9 of 40). Each found many bugs the other did not. Per-finding details are withheld, because several repositories are private and some findings are unfixed.

5.4 RQ4: the Spec review on the other Harness

The recorded Claude Spec reviews raised behavior bugs in 23 of the 30 Specs, and Codex alone raised them in 22. These are sorting-agent counts and were not graded beyond the missed findings above. A second sorting pass counted 26 for the recorded reviews, so the margin is uncertain, but in neither count does Codex reach the recorded review's total. Swapping would also forfeit the recorded review's own catches. RQ4 fails.

5.5 Post-hoc follow-ups

Both follow-ups ran on 12 October 2026 with the same script, isolation and grading prompts as the arms they extend. Plans ran 02:16–07:43 UTC and Specs 02:30–03:08 UTC, one session at a time per Harness. All 24 sessions exited normally, and none changed its repository.

Claude at max on Codex's plans. Claude at high had stopped only two of Codex's plans, while the passing forward arm had run at max, so we reran the reverse at max. It stopped three plans and the grader upheld two, which meets the bar exactly. Both upheld stops (plans #463 and #465) identified decisions under which an interrupted step would be reported or cleaned up wrongly. Fresh Codex had stopped both plans for the same reasons, and Claude at high had stopped #465. The third stop, #499, named a real but minor issue that a builder would settle in passing. On Codex's plans, then, every arm's correct stops fall on the same two plans. Claude at max adds one fewer wrong stop (1 vs. 2), not a new catch. This is the only comparison in which the other family came out ahead, and it did so on precision by a single plan. Both passing plan-check arms used Claude at max as the checker, which suggests that the checker's capability matters more here than its family.

Fresh Codex on Codex's own Specs. Codex had run five Spec reviews over four Specs (#432, #465, #486, #427), between 6 and 9 October. The second review of #486 saw the first's fix, so only the first is replayed. Each session used the review skill its recorded review had run. Fresh Codex found real missed bugs in 2 of the 4 Specs, with 3 findings, all graded real and none wrong (Appendix B). One finding was half of bug #603, which later reached main and was fixed. Neither RQ1 reverse arm caught #603 in the Ticket PRs it came from. Four Specs cannot meet the bar of 6 in 30, so this is a pilot. It is, however, exactly the pairing a same-Harness second opinion would run in production today.

6.1 Freshness, not family

At every checkpoint, a fresh same-model session caught as many of the target defects as the other family, or more. In RQ1 the fresh control caught everything Codex caught. In RQ3 the same-family fresh session reached more features than Codex. The only advantage the other family showed was precision: fewer wrong findings in RQ3, and one fewer wrong stop in the plan follow-up. The prior work on self-preference [1, 2] concerns judgments of quality. Our results suggest that for finding defects, what dominates is whether the reviewer watched the code being written, not which family it comes from. A practical consequence is that the “fresh session” control should be the default comparison for any proposed cross-family reviewer. Without it, gains from independence are credited to diversity.

6.2 Placement matters more than identity

The same reviewers that added nothing inside a Run found real bugs in more than half the features after the Spec review. One explanation is that the Spec review is the first time anyone sees all of a feature's Tickets together, and defects between Tickets live there. A competing explanation is that the RQ3 bar (any real missed bug) is looser than the RQ1 bar (a specific known escape). The two pass rates are therefore not directly comparable, and we cannot separate these explanations with this design. The #603 case in the pilot is suggestive. Both Ticket-level replays missed it, while the feature-level second look caught half of it.

6.3 Finding is not acting

The original in-session reviews had raised two of the six escaped timing bugs (#102 and #303). In both cases the authoring agent declined them as decisions for someone else, and the pull requests merged themselves. The fresh arm's four catches are confounded: it changed both the session and the review skill (§7), so the gain cannot be credited to either alone. The two declined findings, however, show a failure that no additional reviewer fixes. The system has since shipped evidence-backed findings. A correctness finding must carry a test; the author must run it; a failing test obliges a fix, and a decline must cite the run [7].

6.4 Model strength

Model strength alone does not explain the RQ1 result, because the stronger model failed in the second seat too. For the plan check it may matter more. Both passing configurations used the strongest checker at its highest effort, whichever family wrote the plans.

Same-family grading. Every grader was a Claude agent, and Claude is one of the compared families. Labels were shuffled and identifying lines removed, but Codex and Claude reports differ in style, so a grader could infer the arm. The author chose agent grading, so this risk belongs to the design.

One session per cell. Run-to-run variance was not measured. Ties and one-item differences, including the plan follow-up's margin, are within noise. In particular, we cannot say whether Codex's 13 RQ3-unique bugs exceed what a second fresh Claude session would add by chance.

Confounded treatment in RQ1. The fresh arm differed from the original reviews in both session freshness and review-skill version.

Late arm. The RQ3 fresh-Claude arm was added after the Codex results were known, although its procedure matches the Codex arm's.

Construct validity. Partial catches do not count. Counting them would change no outcome. RQ1 and RQ3 use different bars by design.

External validity. The subjects come from one system and six repositories by one developer, with specific model versions as of October 2026. Production has since moved to Codex, where only four Specs existed, so the RQ3 result on Codex-written features rests on the pilot in §5.5.

Small samples. Six and three bugs, 20 plans per direction and 30 Specs can show only large effects. The bars were set with that in mind.

Before crediting a gain to a different model family, run the same model in a fresh session. In 183 preregistered replays, a fresh same-model reviewer matched or beat the other family's catches at every checkpoint. The one place a second look clearly paid, after the feature-level review, paid for either family. thirdshift will add an opt-in second opinion at that point only, with the reviewer's Harness, model and effort configurable. Our current lean is a fresh session on the Spec run's own Harness. A planned look-back will count real fixes over roughly ten production Spec runs from the reviewer reports the system already keeps. That is the test the replay could not run.

Antigravity 1.3.3 changed how its -p flag takes the prompt. The replay script was adjusted, and the same incompatibility in thirdshift's own adapter was filed as issue #658 and fixed in pull request #661.