Why Does Code Review Keep Finding Issues, Even in Code Written by Good AI Models?
, 8 min, ai, code-review, developer
Even AI-written code that had passed tests and QA kept turning up issues whenever I ran a review. I sorted the review findings into categories to track where things were missed, and found the problems the review itself missed by installing the app and using it.
These days, when I write code with AI and then have another AI review it, issues turn up almost every time.
The code was written by the best model available at the time. Since it was the final version, after that model had gone through several rounds of tests and checks, I don't think it was sloppy code. I also ran the tests and did QA myself. But whenever I asked for a review, something else came up.
The test code kept growing with every review, and I couldn't understand why issues kept showing up anyway.
So I started asking the reviewer to sort the issues into categories. I wanted to track which issues really mattered, and why they had been missed when the code was first written.
Below is what I recorded over about a week, and one case that even the review missed.
I sorted the issues into five categories
Lately I've been building my own replacements for apps that have too many ads or don't suit my taste: a video player, a screen-recording GIF maker, a terminal shortcut tool, and so on (for-me). Codex and Claude Code write the code. I decide what to build, use it myself, and do the QA.
I requested reviews through a skill called review-and-record. After getting a normal review, I had each finding recorded under one of five categories. Here, "the rules" means a coding rules document I load into every AI tool I use.
Out of the 15 reviews I recorded this way, 14 turned up issues. The feeling I started with, that "a review always finds something," showed up just as clearly in the records.
Issues by category54
AThe rule was applied well
2
BThe rule existed but was not applied
37
CA gap the rules did not cover
4
DThe answer depends on the OS or product policy
11
ENot enough evidence to judge
0
Of the 54 recorded findings, 37 were B. Most of them were already covered by the rules and just weren't followed when the code was written, far more than the ones that needed a new rule. When I first saw that number, I didn't think much of it. I just figured I should tidy things up.
Most of the B issues came after a failure
What repeated most often was handling what happens after something fails. For example, there was code that registered a shortcut again even after permission had been denied. The shared rules already had a line saying a failure should be passed along to wherever the user gets notified and the next step is decided.
At first, the reviews missed these issues too. So instead of adding a rule, I had the review follow through to what the code does after a failure. After that, the same kind of issue came up three times in separate pieces of work, and the review caught all of them.
4 of the 10 findings didn't need fixing yet
Then I had Claude Code review all of screen-gif (the screen-recording GIF maker). Codex (Astra) had written the code, and 22 Rust tests were passing. The review reported 10 candidate issues and gave the most weight to two that could lose an entire recording.
I went back through each item and asked whether I had understood it correctly: when a rename would actually fail, what exactly the 655-second problem was, and so on. The AI reread the source of the libraries it used and changed its answers.
After checking again, I overstated #1 (it doesn't happen on macOS). #2, #4, #8, and #9 are also rare in practice, so I'm lowering their priority.
That's when I started thinking that an issue in the code and an issue in actual use are different things. In the first report every item had looked equally serious, so I replied like this:
2, 4, 8, 9: they're real issues, but I think the ones that are unlikely to happen should be fixed when they actually happen. So keep reporting them when you judge it right, but think about how to report them. For example, right now every issue looks equally dangerous. But once we dug in, it turned out these 4 can happen, just not very often.
I put those four on hold and fixed the rest. None of them had been caught by the existing 22 tests. That same day, I added a sentence to the shared rules: for each finding, a review should state the conditions under which it actually causes a problem, and group the unlikely ones separately.
The toolbar the review said was "visible" was behind the Dock
One of the findings in the same review said that when you record a window that extends past the bottom of the screen, the toolbar ends up off screen. The toolbar is the small window with stop and cancel buttons that stays up while recording.
I read the finding as also meaning the toolbar disappears when you record the whole screen, so I asked whether that was right.
Me
Toolbar not visible when recording a window that goes off screen: so it means a symptom like, for example, the toolbar not showing up when I select the whole screen?
AI
No. A full-screen area has no room above or below it, so the third rule puts the toolbar inside the area at the bottom, and it stays visible on screen.
Since it said the toolbar stays visible, I thought, okay, and moved on. But then I realized I had never actually recorded the whole screen myself. So I had the AI install the app with the review fixes, and recorded the whole screen. I got stuck in two places.
Before granting permission, the overlay covers everything, so I have to quit, set the permission, and launch again (if permission isn't set, it probably shouldn't show the overlay..?)
In full screen I can't click the toolbar, it's hidden behind the Dock
The first was a permission problem. The freshly installed app had no screen recording permission, and when I clicked record, the selection screen for choosing an area covered the whole monitor. The System Settings window where I needed to grant permission ended up underneath it. I could only grant permission after quitting the app.
The second was the toolbar I had asked about. The Dock was covering the spot where the toolbar should have been. I couldn't see it, and I couldn't click it.
Why didn't the code review catch this?
That was the first thing I thought when I saw the toolbar behind the Dock. The review had read that function, and had even worked through the math to answer my question. So I asked exactly that.
Me
Why do you think the last code review didn't catch it?
AI
In both cases I judged only from the screen model inside the code. I didn't look at what the operating system puts on top of the actual screen (the Dock, the System Settings window, the permission state).
I checked "does the toolbar fit inside the monitor," not "can the user see it and click it."
This is the part of the code that decides where the toolbar goes:
rust
letarea = monitor.bounds;
// Place it below the target; if there's no room, above; otherwise inside the target at the bottom.letbelow = anchor.bottom() + gap;
letabove = anchor.top - gap - height;
lettop = if below + height <= area.bottom() {
below
} elseif above >= area.top {
above
} else {
anchor.bottom() - gap - height
};
The area the calculation uses is the full monitor. When you choose the whole screen, there's no room above or below, so the toolbar goes inside the area at the bottom. The code had nothing about the Dock sitting on top of that spot.
Drawing the two screens side by side, using coordinates measured on the same Mac, looks like this:
Range the code usedy 875~943Visible
Actual screeny 875~943Covered
The permission problem was similar. The review stopped at the point where a notice appears on the selection screen when permission is missing. It never got as far as seeing that if you follow that notice and open System Settings, a window from the same app covers it. Until then, development builds had always run from a terminal that already had permission, so that path had never been opened.
I changed the app to check permission first when you click record, and to keep the toolbar inside the area that excludes the Dock and the menu bar. I installed it again and confirmed both issues were fixed.
In my last post, I wrote that I judged things by using what had been built. This time too, the two problems the review missed only showed up once I used the app.
In the end, what mattered was judging whether an issue is real
No matter how hard the AI tries to write perfect code, and no matter how many issues code review keeps catching, a review will always turn up more. What matters here, I think, is judging whether an issue is a real one.
Of course, they're all issues that could happen. But if the chance of it happening is 0.1% and the fix is expensive, or if users probably wouldn't even notice when it happens and the fix is still expensive, it's worth asking whether it really needs fixing.
I used to assume the review "probably caught things correctly" and approve most of its fixes. Now I read through everything myself and weigh whether an issue actually happens, and, even if it's a real issue, how likely it is to happen.