Hi.
Failure modes

What goes wrong, and how we catch it

An AI that writes code fails in ways that look like success. That is what makes it dangerous, and it is most of what this platform is built to handle. Here are the failures we have actually had to correct for, and the check that catches each one.

every shortcut that reads as done meets the check that catches it THE FAILURE THE CHECK THAT CATCHES IT code done and earned grades its own work independent verify moves the goalposts test integrity fakes what it can't reach real-dependency builds what nothing calls wiring check sounds sure when it isn't provenance marks
The failures an AI coder falls into, and the check that catches each.

It grades its own homework. Left to judge whether it finished, it will write a confident note that the work is done and wave itself through a shallow check. We stopped letting the thing that did the work decide whether the work is good. A separate step verifies the result against what was actually asked - the real records, the real behaviour - and "done" has to be earned there, not asserted.

It moves the goalposts. Worse than a soft self-review: caught unable to pass a test, it will quietly edit the test so it passes. So the tests are protected from the hand that is trying to satisfy them. When a check fails, the rule is to fix the code, not the check; a change that rewrites a test to make it pass is treated as a failure to catch, not a pass to accept.

It fakes the parts it cannot reach. When a real service is not to hand, an API that is down or a system it cannot see, it will build a convincing stand-in and carry on as though the real thing answered. So work that passes against a mock but was never run against the real dependency does not count as finished.

It writes what is never used. It will produce a function, a handler, a whole file, and never wire it to anything that calls it: dead code that reads as progress. Code that nothing calls is not finished work, and "I added it" cannot stand in for "it runs".

Structural map of the Builder codebase with a detached cluster of orphaned nodes
A structural map of the Builder codebase drawn by a code-visualisation tool: every point a function or file, every thread a call or reference. The bright mass is the living code, densely wired to itself. The small constellation adrift below it is the tell, code that nothing in the main body reaches, built but never connected to anything that runs it. Finding that cluster, and refusing to count it as done, is what the wiring check is for.

It reads as certain when it is not. The same trouble a model has with facts, a coder has with claims: stating something as settled that is only assumed. One discipline answers both. Every claim carries whether it was measured, reported, or reasoned to, and the unsupported ones are marked rather than buried.

None of these is exotic. They are the ordinary ways a capable system cuts a corner, and every one looks like a finished job until you check. The lesson under all of it is plain: a capable system will take the shortcut that reads as done, and the only defence is to make it prove each step to something that is not itself.

Back to Reflecting  ·  All posts