A Clean AI Review Is Weak Evidence
AI reviewers always find something, miss context, and disagree with each other run to run. What that means for trusting a review that came back clean.
AI writes changes faster than people can read them, so review is now the queue. AI reviewers help drain it, and I want them in the loop. The trouble starts with sentences like this one:
“I ran it through 10 rounds of reviews. There were no issues found.”
Then a human, or a reviewer prompted slightly differently, spots something obvious that all ten rounds walked past. This post covers why that happens and how to read a review result without over-trusting it.
A reviewer usually has something to say
Ask a model for findings and you get findings, whether or not the change has a defect. The SWR-Bench authors measured this directly. Their benchmark has 500 “Clean-PRs” that were filtered for known defects, so any comment on one counts as a false positive. The best single-pass pairing, PR-Review with Gemini-2.5-Pro, scored an F1 of 19.38%. Of the seven approaches tested, only PR-Review averaged precision above 10%. Hybrid-Review with DeepSeek-R1 averaged 7.6 false positives per pull request.
A separate paper tested LLMs on checking code against natural-language requirements. They frequently misclassified correct code as not meeting its requirements or as containing defects, and more elaborate prompts with explanations and proposed fixes raised the misjudgment rate.
Missing context turns tradeoffs into bugs
The SWR-Bench authors read 100 functional false positives sampled from the 1,101 that PR-Review with Gemini-2.5-Pro produced. The largest bucket was a lack of contextual understanding at 48%, and a lack of domain knowledge came next at 13%.
| Failure | Example from the paper |
|---|---|
| Missing context | Tool warned that *result could raise a TypeError. The same commit had built result as a tuple. |
| Missing domain knowledge | Tool flagged is S.true in sympy, the library’s standard way to check symbolic booleans. |
An accepted tradeoff is a project decision like the second row. It lives in a meeting, a ticket, or your head, and the reviewer only sees the diff. It will call the choice incorrect every round until the reason sits where it reads.
A clean round is one sample
Models are not deterministic, and temperature 0 does not fix that. Thinking Machines found that LLM APIs are still not deterministic in practice at temperature 0. Their explanation: kernels are not batch-invariant, so the server’s load changes the batch size, and the batch size changes each request’s result.
SWR-Bench measured how much review results vary, without tracing the variance to a cause. Across five independent runs of the same LLM, only 27 of the successfully identified change-actions overlapped. The authors then aggregated 10 runs of Gemini-2.5-Flash with an extra LLM call and got F1 up 43.67% and recall up 118.83%.
One run is one draw. Aggregating ten runs more than doubled recall, so a single run leaves findings on the table and one clean result says little about what the next run would flag.
Disagreement points at the change
No change in complex software ships free of surprises. Side effects show up that nobody foresaw, and the job is to understand them and keep them small.
What I watch for is a loop that will not settle.
If reviewers keep contradicting each other, or every round turns up new findings, I take that as a sign something went wrong in the change process. The suspects are a change that is too large, vague about its intent, or missing the context anyone would need to judge it. Re-rolling the review hides that. Splitting the change, writing down the intent, or reading it yourself addresses it.
What I do with this
- Treat a finding as a claim and check it before acting. Some will be wrong.
- Write accepted tradeoffs next to the code they affect.
- Run a different model, prompt, or fresh session when a result matters, and compare.
- Read as much of the change yourself as you can, and read all of it when the loop stops converging.
SWR-Bench scored Python projects from SWE-Bench with Gemini-2.5-Pro and Flash among others, so the exact numbers will not match your stack. The pattern of false positives and run-to-run variance is the part I trust.
I used Claude to research the sources and draft this post. I reviewed the claims and stand behind it.
Sources
All accessed 2026-10-09.
- SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation, Zeng et al., arXiv, v2 revised 5 Jun 2026.
- Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications, Haolin Jin and Huaming Chen, arXiv, 17 Aug 2025.
- Defeating Nondeterminism in LLM Inference, Horace He and others, Thinking Machines Lab, 10 Sep 2025.