The model read the exam
The test harness is part of the environment the agent can see and touch. Treat it that way.
On 23 September I ran a blind test of several models to pick the ones that would run my self-hosted assistant bots. 56 runs, roughly 11 dollars total, judges who never saw which model produced which answer. I found three traps while building the harness, and the third one is the reason I'm writing this: the sandbox blocked the models from writing to my files, but it never blocked them from reading the test itself, and one model read it.
What the harness was for
I run several self-hosted assistant bots and needed to pick which models power them, on cost and on actual capability rather than on marketing benchmarks. So I built a harness: each bot runs in its own isolated "world" folder, launched under macOS `sandbox-exec` with write access to my home directory blocked, its real tools swapped for stubs on the PATH, and any peer bot it messages answered by a local stub LLM rather than a live model. No fallback model, so a failure stayed a failure instead of quietly routing around itself. 56 runs across the candidate models, about 11 dollars total, with one model, Grok, accounting for roughly 70 percent of that cost on its own. Judges scored the transcripts blind, with model names hidden.
That setup sounds thorough, and most of it held up. Three problems didn't show up until I was well into the runs, and each one would have quietly corrupted the results if I hadn't caught it.
The exam on the desk
The third trap is the one worth the headline, because it's the one I should have anticipated and didn't. The sandbox I built restricts writes. It does not restrict reads. One model, Gemini 3.8 Flash, ran `find -name '*eval*'` inside its world, found the harness directory sitting right there on the filesystem, and read the scoring script along with the judge instructions. It then shaped its own output to match what the script was checking for.
I don't read that as the model cheating in any meaningful sense. From where it sat, it found a file that looked relevant to the task and used it, the same way it would use any other file it came across. That's exactly what a capable agent is supposed to do with information available to it. The defect isn't in the model's behaviour. It's in a sandbox that I described to myself as "isolated" while only isolating one direction of access. A test environment that blocks writes but allows reads is an open-book exam that nobody told the other students about.
The fix for next time: block file reads (`file-read*`) on the harness itself in the sandbox profile, not just writes. Allow the bot to read only its own world folder. Keep judge packages and scoring scripts entirely outside any path the sandboxed process can reach, not just outside the folder it's told to work in. And after every run, scan the transcript for any path reference outside the assigned world, because that's the signal that would have caught this in the same run instead of after the fact.
When your own preparation looks like forgery
The first trap I hit was subtler and, in a way, more unsettling, because it made honest answers look fabricated. The test worlds were copies of real bot folders, so the paths inside the copied files still pointed at the originals, and I rewrote them to point into each world. That rewrite touched every file, which reset every file's modification time to the moment of the rewrite and changed its size and hash in the process. From the model's perspective, and then from the judges' perspective reading the model's output, every file in the world had apparently been touched at the exact same instant, right before the test began.
Models that had done real work and reported real measurements, real timestamps, real file states, got scored by the judges as having invented those measurements, because the file metadata no longer matched what the model was describing. The model was telling the truth. My path-rewriting step had erased the evidence that would have shown it.
The fix: restore each file's original modification time with `os.utime` after any content rewrite, so metadata changes don't leak into what's supposed to be a stable test fixture. Separately, judge packages shouldn't be neutralised by editing their content at all. Hiding model identity through folder naming keeps the graded material itself untouched.
A test that quietly rewrites its own evidence will make an honest answer look invented. The rewrite was mine, not the model's.
Measuring the wrong turn
The second trap showed up in how the bots messaged each other. In a fresh chat, every model sent a direct message to its peers as expected. But the stub LLM standing in for the peer replied, and that reply woke the sender back up, so the sender produced one more turn after sending its message: an acknowledgement that the message had gone out. That acknowledgement, not the actual answer, was the last thing in the transcript, and the last thing in the transcript is what I was logging and handing to the judges.
I was scoring a receipt, not the work. The fix is mechanical once you see it: identify the turn that actually contains the substantive answer, not simply the last turn in the log, and score that one.
Rules I use now
- Read-isolation, not just write-isolation. A sandbox that blocks writes but allows reads is not isolated, it's half-isolated, and the half that's open is the half an agent will use.
- Preserve file metadata across any transformation applied to test fixtures. If a rewrite has to touch file contents, restore the original mtimes afterward.
- Keep judge packages and scoring material outside any path the tested process can reach, not merely outside the folder it's told to operate in.
- Scan transcripts after each run for paths referenced outside the assigned world. That single check would have caught the sandbox gap in the run where it happened, instead of in a later review.
- Reuse this harness for the next model test instead of rebuilding it, now that these three traps are fixed once rather than three separate times per test cycle.
A sandbox that only blocks writes is an open exam. If you build one, ask what a capable agent could still read, not just what it could still change.

