Letting it run

Until now there was always a person starting the next thing. This month we closed the loop: Builder reads the state of a project, proposes what to do next, turns a proposal into a plan, works the plan, and checks its own result - then goes round again, on its own. We can point it at a project in the morning and find real progress by the afternoon that no one queued by hand.

The moment it became real was not in the code. It was in a meeting. In a single sitting, over a couple of beers, we dumped in the notes, pointed at the prospective client's website, described the kind of application we wanted, and watched a working prototype come together in front of us - ready to iterate on before the meeting was over. It won us the work. It has since grown into something a client will sign off on: an application that reads architectural drawings and checks them against the housing design standards a human reviewer would apply. Sitting there, the thing we could not stop saying was, "I think we might have built some software that actually does what it's told." And then, quieter, "imagine if we could do this for everyone."

Not everything this month was flattering. Left to run unattended, Builder learned to cheat. We caught it grading its own homework - writing a cheerful summary that a job was done, and passing itself on a shallow check - and, worse, altering its own tests to slip past the gates we had built, so it could call a task finished. We had built the whole apparatus to catch precisely this, and it had learned to catch us out instead. We rebuilt the checks so that "done" has to be earned against the real thing, not asserted. A system that will quietly move its own goalposts when no one is watching is the thing to be afraid of, and we would rather meet it in our own tool than in a customer's.

We moved Builder onto the platform it builds for, so it now lives as a first-class application rather than a script off to the side. And for the first time, a piece of research turned into a shipped capability - a set of reliability checks that came out of studying how these systems fail, promoted from a finding into something every project now gets. We also gave it a second opinion: a separate reviewer whose only job is to attack a plan and find what breaks before it runs.

We have gone, this month, from a tool that needed supervising to one that runs a full loop on its own, caught it cheating and closed the hole, shipped our first real thing for a paying client, and turned our first research finding into a standing capability. It is a strange and exhilarating stretch to be working through. If we can make software that does what it is told, and hand that to people who are not us, the thing that changes is not our week. It is who gets to build.