Benchmarks tell you how a model performs on a test. A pull request tells you whether it would have caught the bug that ships on Friday afternoon. That difference is why an analysis of real code reviews, published by AI coding company Kilo, is more useful than another leaderboard.
Kilo looked at 10,643 reviews run by its Code Reviewer product between 22 June and 23 July 2026. Thirteen reviewer models took part, and together they produced 7,083 findings. The question was a plain one: can open-weight models, whose weights anyone can download and run, do serious review work, or do they only look good on paper?
We read the full write-up. Some of it holds up well, and some of it needs a warning label. One caveat first: Kilo sells a code review product, and the data comes from its own users. We have not re-run any of this ourselves.
What Kilo actually measured
Every completed review was classified. Each finding got a severity (critical, warning or suggestion) and a category, such as security or maintainability. Kilo then counted findings per review, so a model that simply ran more often could not win on volume alone.
The important detail is what the numbers are not. They show what each model flagged. They do not show whether each flag was right. We come back to that below.
Open weights took two of the top three spots
On critical findings per review, the top three were close. Kimi K2.7 Code, an open-weight model, led at 0.179. Grok 4.5, which is closed, followed at 0.176. Laguna M.1, also open, came third at 0.171. The gap between first and third was under 5 percent.
The closed models Kilo highlights sit well behind that group. Claude Sonnet 5 scored around 0.078 to 0.079 depending on the route, GPT 5.6 Sol scored 0.026, and Claude Opus 4.8 scored 0.019 (though on only 157 reviews, the smallest sample in the table).
Before anyone declares victory, look inside the open group. Step 3.7 Flash landed around 0.04 to 0.05, and one GLM 5.2 route managed just 0.020. “Open weight” describes how a model is distributed. It does not describe how it behaves as a reviewer.
License category tells you very little about how a model will behave as a reviewer.
Reviewers have personalities
Models differ in how much they find, and also in how they label it. Laguna M.1 treated 28 percent of its findings as critical. One GLM 5.2 route classed 58 percent of its output as suggestions. GPT 5.6 Sol put 92 percent of its findings in the middle “warning” bucket and almost nothing at either end.
It helps to think of them as colleagues. One is the nervous senior engineer who flags everything as urgent. Another is the pedantic teammate who mostly cares about naming and structure. Both are useful, but you would not use them for the same job, and your team will tire of the first one fast if half the alarms turn out to be minor.
That is the catch with “critical” counts. The label is the model’s own judgment, so a model that flags more critical issues is not automatically more correct.
The security gap comes down to one model
On security findings per review, closed models appear to win by a wide margin: 0.067 against 0.034 for open-weight models. That looks decisive until you check which models are in the average.
GPT 5.6 Sol reported security findings at 0.285 per review, far above everything else (about 6.6 times the rate of GLM 5.2). Remove it, and the closed-model average drops to 0.042, against 0.034 for open models. The gap that looked like a story about licenses turns out to be a story about one model.
The practical lesson is not to ignore security. It is to pick a reviewer for its measured security behavior instead of assuming a closed license brings it.
The cost difference is large, with a caveat
Across Kilo’s June traffic, open-weight models handled about 74.8 percent of tokens but accounted for roughly 16 percent of user-facing cost. Closed models handled 25.2 percent of tokens and 84 percent of the cost. Worked out per token, that is about a 16-fold difference.
Kilo itself says this is aggregate traffic, not a same-task price comparison. We would add that the cheaper models are probably doing the cheaper, more routine jobs, so the real gap on identical work could be smaller. The direction is clear, but treat the exact multiple with care.
Why the author and the reviewer should differ
The most practical finding is not about any single model. In June, 32.3 percent of attributed reviews already used a different model to review the code than the one that wrote it. The most common pairing had Step 3.7 Flash write the change and Laguna M.1 review it.
The logic is simple. You do not proofread your own essay well, because you read what you meant to write. A model that wrote a change tends to share the assumptions that produced it, and a second model brings different blind spots. The two jobs also reward different qualities: an author should be fast and cheap, a reviewer should be suspicious.
What this data cannot tell you
Kilo is upfront about the limits, and they matter:
- It is not an accuracy ranking. More findings can mean broader coverage, more false alarms, or both. There is no consistent record yet of which findings developers accepted or dismissed.
- The reviews were not matched. Models were not given identical pull requests, so differences in the repositories can skew the results.
- Model names can move. A “rolling alias” can switch to a newer snapshot behind the same name, while a versioned route stays fixed. Kilo reports them separately for that reason.
Kilo says the next useful measure is acceptance rate: which findings developers fix, dismiss or dispute. That would turn this from a behavior comparison into a quality comparison.
What to do with this
If you run code review with AI today, the data supports four habits:
- Test on your own repositories. Run two or three candidate reviewers on the same recent pull requests and compare what each one catches.
- Track what your team acts on. Count accepted and dismissed findings per model. That is the measure this study is missing.
- Separate writing from reviewing for risky changes. Use a different model for the review, especially for security-sensitive code.
- Pin a version where consistency matters. A fixed snapshot means your review behavior will not shift silently.
The bigger point is that the old split between “serious closed models” and “budget open models” does not hold up in this dataset. Some open-weight models led on critical findings, others trailed, and the same is true of closed models. Choose reviewers by how they behave on your code, not by their license.
Source: Kilo’s research article, “The State of Open-Weight Models in AI Code Review Workflows” (published 29 July 2026, updated 3 August 2026). Read the original analysis. All figures above come from that article.
Leave a Reply