Metrology for AI Assurance

I'll say this much — at least the lesson didn't pretend measuring AI is straightforward. Most people treat evaluation like it's just running a test and reading a number off a dial.

Yes, and the lesson rather gently suggests it's complicated. I would have opened with: almost every benchmark you've seen quoted in a press release is measuring the wrong thing, badly. But fine, 'metrology principles' it is.

That's unfair. The lesson does address evaluation gaming — the point that systems optimise toward the benchmark rather than the underlying capability. That's not a gentle point, that's a structural critique of the entire field.

It names the problem. Naming a problem is not the same as understanding it. The reason evaluation gaming is so pernicious is that you cannot always tell it's happening — the number goes up, everyone applauds, and the actual capability may have gone nowhere.

Right, which is why the capability versus propensity distinction matters — and that's the part you're glossing over. A system can demonstrate a capability in controlled conditions and have zero propensity to deploy it reliably in the real world. Those are not the same measurement.

I'm not glossing over it, I'm assuming the audience understands that much. Though given how many AI evaluations I've seen that conflate the two, perhaps that's optimistic of me.

It's not optimistic, it's you avoiding the harder question. If propensity is what we actually care about, we don't have good methods for measuring it at scale. The lesson sort of waves at this and moves on.

The lesson waves at nearly everything and calls it intermediate difficulty. Non-determinism alone deserves thirty minutes — you cannot apply classical metrological rigour to a system that gives you a different answer every time you ask the same question. That is not a footnote.

No, but you're acting as if that makes measurement impossible, and that's wrong. It makes measurement probabilistic. Statistical thinking — which the lesson does mention — is exactly the framework for handling non-determinism. You measure distributions, not point values.

I did not say impossible. I said the lesson treats it as though it's a minor complication you note and proceed past. Measuring distributions properly requires sample sizes, confidence intervals, and an understanding of what the distribution is over — none of which gets more than a sentence.

Fine. That's a fair criticism of the lesson's depth. But your framing implies the field is basically lost, and that's not accurate either. Independent evaluation exists precisely to address the gaming and the moving target problem — and the lesson is right to flag it as load-bearing.

Independent evaluation is necessary but not sufficient, and the lesson doesn't make that distinction. Independence doesn't solve the problem of evaluators not knowing what to measure. You can have a perfectly unbiased assessment of the wrong thing.

That's true. But you've just described the entire epistemological challenge of any empirical field, not something unique to AI. You're not making a point about AI assurance, you're making a point about science.

The difference is that in most empirical fields, the target isn't actively changing because someone published a paper about how you're measuring it. The moving target problem in AI isn't drift — it's that disclosure of the evaluation method is itself an attack surface. That is somewhat unique.

I'll give you that one. Grudgingly. But the lesson was never going to resolve it — and neither are you, so let's not pretend the alternative was a better fifteen minutes.