#evaluation

AI made generating work cheap. Reading it is still a person's job, and the bill for that grows faster than the pile.

Evals are written like unit tests and behave like psychological tests. A century of measurement theory already named the failures teams keep rediscovering.

SRE-Bench spent five thousand expert hours keeping its questions secret. Testing has handled that problem for forty years and calls it something else.

McKinsey counted 1,719 leaders in August. Large firms scaling agents jumped to 40%. Firms moving earnings stayed at 6%. Part of that gap is architectural.

Gartner says 40% of enterprise apps will embed agents this year, and that most vendors are faking it. Both are true because 'agent' stopped being a technical word.

Karpathy reframed LLMs as ghosts trained by imitation of text, not animals that learn the world. The shift from vibe coding to agent engineering is what production has been waiting for.

El País counted us. The handful who actually train models are doing something different than the rest of the AI workforce — and what we're doing is changing in a way the headline misses.

Notes from a ProductTank Madrid we hosted at Shakers, on what changes when AI enters a product team — and what doesn't.