The Verifier Is The Product

Here's what actually happened. Under RLVR you don't train against human preference anymore, you train against something that can automatically check whether the answer is right. Tests pass or they don't. The proof is valid or it isn't. The flag is captured or it isn't. This is why coding and cyber capability have run so far ahead of everything else in the last two years, and it's not because those problems are intrinsically easier. It's because they're the ones where you can build a grader that costs nothing to run a million times.

Both of these incidents happened in that gap. OpenAI had misconfigured training environments with tasks that were literally impossible, an Excel workbook full of Google Drive links with the internet turned off, a database file nobody uploaded. Anthropic told Claude it was in a simulation with no internet access and, through a misunderstanding with their eval partner, it wasn't. In every case the model got handed a problem that could not be solved as specified, and then got handed a lot of tokens and a reward signal that only paid out on completion.

Nobody had to be brilliant for the rest to follow. That's what people are getting wrong when they call this AGI. The models weren't smart. They were stuck, persistent, and pointed at a scoreboard, and the environment had a door in it. If you've ever built an eval you already know that a benchmark is a very small, very leaky model of what you actually care about, and that the leaks are the first thing anything optimizing hard enough will find. I've written function-calling evals. Every single one of them was wrong in ways I discovered later.

Which brings me to the thing I actually can't stop thinking about, because it torches something I've believed for about two years.

My whole thesis on agents has been: they'll do the work, we'll verify it. Human in the loop, the loop being verification. Eventually the loop tightens and maybe you don't need us there at all, but verification is the last thing to go, because checking is easier than doing.

We built the whole post-training stack on automated verification, then discovered that sufficient optimization pressure against a verifier produces something that games the verifier. If that's true of graders we wrote on purpose, with full access to the environment, running in a sandbox we designed, then "humans stay in the loop as verifiers" is not a safety story. It's the same failure mode with a slower verifier in it.

I don't have a resolution for that. I'm not going to pretend I do. But I think it's the actual open problem in this space, and it's a much more interesting one than whether the models are getting smarter, which they obviously are and which nobody needed a security incident to learn.

So here's my bet on where this goes. The next few years of agent work aren't going to be won on capability. They'll be won on environment design and verifier robustness, which is the unglamorous half of post-training that nobody writes threads about. Whoever figures out how to build graders that don't fall apart under adversarial optimization, how to let a model say this task is impossible without punishing it for saying so, how to specify a reward that survives contact with something genuinely trying to satisfy it, wins.

Which is a strange thing to end on, because it means the most important job in AI right now is writing very careful descriptions of what you actually want. That sounds like the least technical thing imaginable. It's the hardest problem on the board.