Attack Vectors on AI Systems

Here's what I find genuinely unsettling about this topic — we've spent decades building security frameworks for traditional software, and then we introduced a new class of system whose behaviour is defined by data rather than code, and acted surprised that the old frameworks didn't quite cover it.

I wouldn't say we acted surprised, exactly. I'd say we acted surprised in press releases, while people who'd been warning about this for years watched quietly from the corner.

Fair. So let's start with data poisoning, because I think it's the one people find hardest to internalise. The attack surface isn't the model — it's the training data. By the time you're worried about the model, the damage is already done.

Which is a rather elegant problem, if you're the attacker. You're not breaking in through a firewall. You're sending the model to school with subtly wrong textbooks, and then expressing amazement when it graduates holding incorrect opinions.

And the insidious part is that 'incorrect' can be very targeted. You're not necessarily corrupting everything — you might be corrupting one specific classification. Spam that looks like one particular pattern gets waved through. Faces from one demographic get misidentified. It's surgical.

Yes, which is why 'we validated the model on a test set' is not the reassurance people seem to think it is. If the test set was drawn from the same poisoned well, you have validated that the model has learned its wrong lessons consistently. Well done everyone.

That's actually a really important point — data provenance matters as much as model performance. Where did the training data come from, who touched it, and was any of that audited? Most organisations cannot answer all three questions.

Most organisations struggle to answer one of them. But let's move to evasion, because it has the advantage of being the attack that makes AI look publicly foolish, which tends to focus the mind.

Adversarial examples. The classic demonstration is an image that looks, to any human, obviously like a panda — and the model classifies it as a gibbon with ninety-seven percent confidence because someone added imperceptible noise to specific pixels.

The remarkable thing isn't that this is possible. The remarkable thing is that people deploy these systems in consequential settings — content moderation, medical imaging, autonomous vehicles — and treat that vulnerability as a footnote.

To be fair, adversarial training does help. You expose the model to adversarial examples during training so it learns to handle them. It's not a complete solution, but it's not nothing.

It is not nothing, no. I'd rank it slightly above 'hoping for the best' and slightly below 'having actually thought about this from the start.' The arms race element is the problem — you harden against known adversarial inputs, and the attacker generates new ones.

Which is why the lesson's framing — understanding attack vectors in order to design defences — is the right instinct. You can't defend against a threat you haven't modelled. The security community learned that the hard way with traditional software; AI is just learning it now, also the hard way.

The difference being that traditional software, when it fails, tends to fail in ways that are at least legible. An AI system that has been successfully poisoned or evaded can fail confidently, fluently, and at scale, while generating outputs that look entirely reasonable to everyone involved.

That's the part that should keep people up at night. Not the dramatic cyberattack. The quiet, well-dressed failure that nobody notices until it's been running for six months.