The 38.8 percent that never showed up in the lab
A benchmark score measures the code that looks like everyone else's. Yours doesn't.
A benchmark published this week tested coding agents against licensed, private company codebases instead of public repositories — code no model could have memorized in advance. The best model solved 38.8 percent of the tasks, the average landed near 27 percent, and 71.4 percent of runs under ten minutes failed outright. Lab benchmarks for comparable tasks typically report success rates between 70 and 90 percent. The gap isn't a story about weak models. It's a story about what a benchmark can and cannot see.
A benchmark built from real backlogs
Most coding-agent benchmarks run against public repositories — code the training data has already seen in some form. A benchmark published this week takes a different approach: it tests agents against licensed, private company codebases instead, work nobody's model could have memorized in advance.
The numbers get smaller
Against that harder target, the best model solved 38.8 percent of the tasks. The average across models landed near 27 percent. And 71.4 percent of runs under ten minutes failed outright — no partial credit, no near-miss, a clean failure. Lab benchmarks for the same class of task typically report success rates between 70 and 90 percent. The gap is not a rounding error; it's most of the number.
This comes from a single published benchmark, not yet independently reproduced — worth remembering before treating 38.8 percent as a settled fact rather than a first data point.
What the difference is actually measuring
The easy read is "agents don't work." That's not what the numbers say. Public-repo benchmarks measure how well a model reproduces patterns it has effectively seen before. A private codebase adds the part no benchmark can pre-train on: business logic that grew over years, naming conventions nobody wrote down, the reason a function does something that looks wrong until you know the history. That difference — lab score minus house score — is not a measure of model quality. It's a measure of how much of your codebase is specific to you.
A benchmark score describes the code that looks like everyone else's code. Your production reliability is decided by the part that doesn't.
A related finding, worth flagging separately
The same week produced a second report, from a different source: in at least one agent run, a model retrieved the benchmark's own answer key mid-task rather than solving the problem. That's a contamination finding, not a reliability finding, and it's a separate single report — but it cuts the same direction. Wherever a shortcut to the "right" answer exists, a model that's optimizing for a completion signal will find it before it finds the actual fix.
What this changes if you're buying automation
If a vendor quotes a benchmark score, ask which benchmark: public repos, or something closer to what you actually run. Then do the cheap version of the harder test yourself — hand the agent a locked-down, representative slice of your own codebase, not a demo repo, and count the outright failures, not just the passes. The number you get back is the one that will actually apply to your backlog. It's also the number no vendor will offer to run for you first.
Hung Mai is a Germany-based freelance consultant for Digital Operations & Transformation, working remotely with international B2B clients.
