My AI reviewers were right about almost everything. Not one finding was worth stopping the merge for.

I did the responsible thing. When I wanted my AI code review to be more than one model's opinion, I built a panel: several different model families, each reading the same diff independently, each emitting findings with a file, a line, a claim, and a severity. If one model missed a bug, another would catch it. If they all agreed, that was signal. A jury, not a judge.

Then I did the thing almost nobody does to an AI reviewer. I audited it, by hand, at the source. I took twenty-five findings the panel had stamped blocker, its highest severity, the one that means stop the merge, and checked every one against the actual code at the exact commit it reviewed.

Two numbers came back, and they do not belong in the same system. Ninety-two percent of the findings were true. And zero of them were worth blocking a release for.

Detection and prioritisation are different jobs. My panel had aced the first and had never once been graded on the second. Here is the whole teardown, with the receipts.

Right about everything, useful about nothing

The cases I audited were clean, already-merged code that a human had approved and shipped. So "true" does not mean "caught a bug that took down production." It means something more unsettling: the panel looked at code everyone had signed off on and pointed at twenty-five real things. A missing test on an error path. A symbol no caller reaches. A validation quietly widened past its contract. Each one, traced to the line, was actually there. The models were not hallucinating. They have very good eyes.

And I would have shipped every one of those diffs anyway, because not one of those true findings was the kind of thing you hold a release for. They were the software equivalent of a proofreader who circles every "which" that should be a "that." Correct. Exhausting. Not a reason to stop the presses.

The objection I have to clear first

If you are sharp, you already have the counter loaded, so let me make it for you, harder than you would: of course nothing was merge-blocking. I sampled findings on code that had already passed review and shipped. A corpus of already-merged diffs is a corpus selected to contain almost no genuine blockers. The zero was baked in.

That objection is correct, and it does not save the panel. Here is why.

Grant the premise completely. Assume the corpus contains no blocker-grade defects at all. The panel still labelled all twenty-five findings blocker. That is a fact about the label, not about the code. A severity label that fires at essentially 100% on a corpus you have assumed holds no blockers is a label that carries no information, and a gate is a machine that consumes exactly that label to decide whether to stop your merge. The survivorship objection does not rescue the reviewer. It convicts the label a second time, on independent grounds. Whether the code was blockable and whether the panel can tell are two different questions, and the audit only sharpens the second.

Review is two jobs, and the industry grades only one

"Reviewing a change" reads like one task. It is at least two, and a model can be excellent at one while being uncalibrated on the other.

We benchmark detection, because detection has a scorable right answer. Then we deploy the thing as a gate, which is a prioritisation job, and act surprised when it cries wolf. I had built an excellent smoke detector and wired it to the fire alarm for the whole building.

Everything below is an attempt to measure the second column honestly.

The audit, and how to run it on your own panel

A claim about reviewer quality is worth exactly as much as the audit behind it. This protocol is not complicated and it is not specific to my system, so take it.

The adjudication protocol was four rules, each of which exists to stop me flattering myself:

1. Census, not sample. Every blocker in the set, so there is no cherry-picking to argue about.
2. Judge at the commit, not at HEAD. Every finding checked against the tree as it stood when the panel reviewed it. A later commit that quietly added the missing test does not get to make a true finding look false.
3. Evidence or it did not happen. Every verdict carried a file:line citation or it was discarded.
4. Then attack the verdicts. A second, independent pass existed only to overturn the first, with a bias toward refutation. Its job was to kill my own conclusions. It overturned several. That is the point.

Two axes, held apart the entire time: is the finding true (a fact about the code, checkable) and would it block the merge (a judgment about consequence, the thing a gate actually needs). Conflating those is the original sin of AI review.

The receipts, and a name for the gap

Line the panel up against the only bar the people who actually ship this ever published.

That third row deserves a caveat rather than a claim, because precision and effective-false-positive rate are different axes and a number on one does not locate you on the other. BitsAI-CR (Sun et al., FSE 2025) is a reviewer ByteDance actually runs in production at roughly 75% precision. Whether it clears Google's actionability bar is not something its precision figure can tell us, and I am not going to pretend otherwise.

The Google line reorganised my thinking because of a single word. An effective false positive is not a finding that is factually wrong. It is a finding the developer does not act on. Sadowski and colleagues (Communications of the ACM, 2018) put it plainly: a check earns its place only if developers feel it is pointing at an actual issue. Precision is not where this is won.

This gap between looking right and being worth acting on is the whole essay, so let me name it, because a quantity without a name cannot be tracked.

The actionability gap. Δ = P_true − P_act

where P_true is the share of emitted findings that are true, and P_act the share a developer would act on. My panel: Δ = 0.92, and combining the two Wilson bounds conservatively puts it somewhere in [0.62, 0.98]. A near-perfect detector, near the maximum possible gap, even at the friendly end of its own error bars.

P_act is just Google's bar seen from the other side: their effective-false-positive rate is 1 − P_act. Measure Δ on your panel and you are measuring the distance between the review you can benchmark and the review you can trust.

Any agreement score built this way grades the silence

Before the audit, I trusted a number. A per-review reliability score, meant to tell me how much to believe the panel's agreement, built on a standard inter-rater agreement statistic, the kind you reach for without thinking.

It was lying, and the mechanism is not a quirk of my implementation. It is a property of the statistic that any multi-model panel measuring agreement walks straight into. Picture the agreement between two reviewers on one file as a 2x2 count:

                 reviewer B flags   B stays silent
reviewer A flags        a                 b
A stays silent          c                 d      <- d is the trap

Most files in a diff are flagged by nobody. That is correct behaviour; a reviewer should be quiet about the code that is fine. But those files pile into cell d, and the standard statistic counts a big d as agreement. Three models all saying nothing gets scored as three models agreeing. So the reviewer that did the least work earns the highest reliability.

Now put exact numbers on it. Hold two reviewers at a fixed 85.8% raw agreement, so their actual error rate never changes, and vary only how rare the flagged class is. Gwet's AC1, the statistic I had shipped, climbs. The proportion of positive agreement, which by its formula never touches cell d, collapses.

p_pos = 2a / (2a + b + c)        <- cell d never appears

That last line is the finding, and it generalises past my panel: this family of agreement statistics converges on the fraction of files nobody looked at. Wongpakaran and colleagues (2013) tabulated the same collapse in a clinical setting more than a decade ago. The proportion of positive agreement has sat in that literature since the 1970s and refuses to be fooled, because cell d never enters the formula.

One point of precision, since the whole essay is about not overclaiming. A reviewer that flags nothing lands a = 0. If its counterpart flagged anything at all, p_pos is exactly zero, which is the right answer. If neither ever flagged anything, the ratio is undefined, which is also the right answer: there is nothing to agree about.

Run this on your own panel tonight

Three lines of arithmetic, no new model:

For each pair of reviewers, over the reviews you already log:
    a = files BOTH flagged
    b = files only the first flagged
    c = files only the second flagged
    p_pos = 2a / (2a + b + c)   # the files neither flagged never enter

Average p_pos across pairs, and compare it to the agreement number you
report today. The gap between them is how much of your current score
is being paid by silence.

If that gap is large, your quietest reviewer is quietly inflating your confidence, and it is doing it most on your biggest diffs.

The tell: the score was tracking diff size, not defects

Once I suspected the gauge, I checked what it actually correlated with. Not defectiveness. File count.

Across the sweep the correlation with diff size was +0.74, for exactly the cell-d reason: a bigger diff has more files nobody flagged, more silence, more phantom agreement, a higher score. The number I trusted to say "how reliable is this review" was, to a first approximation, answering "how big was the diff."

The 0.858 in that table is my own measured score at 23 files. It is not the same quantity as the 0.858 ceiling in Figure 2, which is a derived limit. The collision is a coincidence, and an unhelpful one, so I am flagging it rather than letting you assume I derived one from the other.

A metric that moves confidently with the wrong variable is worse than no metric, because you act on it.

Nobody can do this: not humans, not the standards

The obvious verdict here is "the models are bad at severity." I went looking for how much better humans are, braced to be embarrassed. I was not.

Sit trained human evaluators in front of the same problems and ask them to rate severity, and their ratings agree at a pairwise rank correlation of about 0.23 to 0.31. Barely above noise, and it holds whether they are usability specialists or graduate students; expertise does not rescue it. That is Hertzum and Jacobsen (Int. J. Human-Computer Interaction, 2003), replicated across three separate evaluation methods.

The vulnerability world spent years building standardised severity scales precisely to remove human disagreement. Koscinski and colleagues (ACM CCS 2025) pointed four of the best-known at the same 600 vulnerabilities and found they agree with each other at approximately chance.

Not because any of those scales is broken, but because each quietly optimises a different definition of "severe," and nobody agreed on the definition first.

So the thing my panel is bad at is a thing humans are bad at, and a thing committees of experts building formal scales are bad at. Severity is not an accuracy problem the models will grow out of. It is a genuinely hard, genuinely unsolved calibration problem, and it already has a name in the static-analysis literature: Actionable Warning Identification, a field that long ago split its output into classification (throw away the unactionable) and ranking (order what remains). I was not the first person here. I had just wandered in without reading the sign.

The traps nobody puts on the brochure

If you run a panel, here is where the bodies are buried.

You are probably optimising precision when the gate lives on actionability

Different axes, and pushing the first does nothing for the second. Every hour I could have spent raising factual precision would have moved P_true from 0.92 toward 1.0 and left Δ almost exactly where it was, because P_act is the term doing the damage.

Multiple lenses inflate the count

My reviewers read each diff through several lenses, and they cheerfully report the same defect more than once. My twenty-three true findings collapsed to fifteen distinct problems once de-duplicated, a 1.5x inflation, with the worst single defect counted five times.

Confidence points the wrong way on absence

The most common failure I saw was the panel asserting something did not exist ("there is no test for this") while looking at a diff that could not possibly show it the whole test suite. The research literature is brutal on this: when a model judges against evidence it was not fully given, it produces more false negatives than false positives, and it is more confident when it is wrongly saying "no" than when it is wrongly saying "yes." So the intuitive fix, "weight findings by the model's confidence," would systematically weight the wrong ones up.

A reviewer that is right often, wrong quietly, and confident on its absences is not a safety net. It is a confident liar in the costume of a precise one, and that is the one thing worse than an honest "I am not sure."

What this establishes

Four claims, stated plainly, because a teardown that only lists what broke is a confession rather than a result.

1. A severity label can be factually excellent and informationally empty. 92% true, 0 of 25 actionable, and the label fires at the same rate whether or not a blocker exists. Truth and actionability are separable properties, Δ is the distance between them, and mine sits at 0.92.
2. This family of agreement statistics converges on the silence rate. Not as an observation about my implementation but as exact arithmetic, shown in Figure 2. Any panel scoring agreement this way is partly measuring how much code nobody looked at, and the effect is strongest on the largest diffs, which is precisely where you least want to be misled.
3. The inflation is structural. Multiple lenses multiply one defect into several findings at roughly 1.5x here and 5x at the worst. Any threshold fitted to a raw finding count is fitted to a number that grows with reviewer count rather than with risk.
4. Severity calibration is not a model-capability problem. Humans fail it at 0.23 to 0.31, four standardised scales fail it at approximately zero, and the gate a model would need to clear starts at 0.61. Waiting for a better model is not a plan.

The fix: rank, do not gate

The way out is mostly a change in what you ask for, not a bigger model.

Stop asking the panel for a verdict. Ask it for an order. The named field already made this move: separate classification (drop the unactionable) from ranking (order by how much it matters). A gate that fires on a count of blockers was always going to lose, because "blocker" was never a calibrated label. A gate that surfaces the top findings and lets a human draw the line asks the panel to do the thing it can do, sort, instead of the thing it cannot, decide.

Fix the gauge before you fit the threshold. The metric that rewarded silence has a peer-reviewed replacement I had been ignoring: the multi-rater proportion of specific agreement (de Vet et al., J Clin Epidemiol, 2017 and 2018), which by its very formula does not count the case where nobody flagged anything. Pure arithmetic, no new model, waiting in the statistics literature for someone to point it at LLM panels.

Track Δ, not precision. Precision is the number you can benchmark. Δ is the number that tells you whether the benchmark means anything. They are cheap to measure together and they diverge in exactly the situations that matter.

What changed, and what would prove me wrong

Three things changed in my own system as a result, and none of them required a better model. The reliability score now reports the proportion of positive agreement alongside the old statistic, so the gap between them is visible on every review. The gate no longer fires on a count of blockers; it ranks and surfaces, and a human draws the line. And Δ is now computed per review rather than discovered once in an audit.

Here is the uncomfortable part, stated plainly, because in a domain whose whole value is trust the method is the argument. The audit above was machine-adjudicated, run in two adversarial stages, but it was a machine judging a machine. The severity call rested on a single rater. The sample concentrated in one part of the codebase. Those are real limits, and they are why the numbers above are a sharp, well-instrumented estimate and not a verdict.

So here is the study that settles it, and the result that would falsify everything I just argued:

• The run. ~30 blocker findings, drawn across commits rather than concentrated, each adjudicated by two independent senior engineers, blind to the panel's label, disagreements resolved by discussion with the inter-rater agreement reported.
• The pre-registration. The scoring rule and the merge-blocking definition are fixed before anyone looks, and findings are de-duplicated by defect identity before a single count is taken.
• The falsifier. If that human jury finds a materially non-zero rate of genuine blockers among the panel's blocker-labelled findings, then the 0-of-25 does not generalise, Δ is smaller than I claimed, and the panel deserves more trust than this post gives it. I will publish the number either way.

That is the difference between a field note and a press release. I would rather hand you a result with its falsifier attached than a rounder one with the exits quietly sealed.

One last thing, and it is a confession rather than a recommendation. The deepest cut in all of this was not the miscalibration. It was that I already owned the fix. The honest agreement score, a paradox detector, a calibrated floor: all built, all sitting one layer away from the decision, all wired to nothing while the gate ran on a hardcoded default. The bottleneck was never measurement. It was connecting what I had already measured to the choice it was supposed to inform, and that is a far more human failure than a bad statistic.

Generation is nearly solved. A model will find your bug. Trust is the unsolved part, and a reviewer you cannot calibrate is not trust, it is a coin flip that happens to sound authoritative. Knowing exactly what to measure, and refusing to ship a gate until the gauge is honest, is the whole job.

Further reading

• Sadowski et al., Lessons from Building Static Analysis Tools at Google, Communications of the ACM (2018), and Tricorder (ICSE 2015): the "effective false positive" and the sub-10% code-review bar.
• Hertzum and Jacobsen, The Evaluator Effect, Int. J. Human-Computer Interaction (2003): why human severity ratings agree at ~0.23 to 0.31.
• Koscinski et al., Conflicting Scores, Confusing Signals, ACM CCS (2025): CVSS / EPSS / SSVC / Exploitability Index over 600 identical CVEs.
• Landis and Koch, The Measurement of Observer Agreement for Categorical Data, Biometrics (1977): the 0.61 substantial-agreement threshold in Figure 3.
• The Actionable Warning Identification literature: the field that separates classification from ranking.
• Sun et al., BitsAI-CR (FSE 2025): a deployed LLM reviewer at ~75% precision.
• de Vet et al., the multi-rater proportion of specific agreement, J Clin Epidemiol (2017, 2018): the agreement statistic that does not reward silence.
• Wongpakaran et al. (2013): the tabulated collapse of Gwet's AC1 under skewed marginals.