Adversarial AI Techniques

I want to start with something that I think gets buried in how this topic is usually taught. The genuinely unsettling part of adversarial attacks isn't the math. It's that a model can be almost perfectly accurate on real-world data and then completely fall apart when someone adds noise that a human wouldn't even notice.

Yes, and somehow people find this surprising. The same community that spent years telling us these models were essentially doing perception has now discovered they are, in fact, not doing perception at all. I am sure that revelation was uncomfortable.

That's a bit uncharitable. The capability was real. The brittleness was a genuine discovery, not negligence.

Is it uncharitable? FGSM — the Fast Gradient Sign Method — was published in 2014. It works by taking the gradient of the loss with respect to the input and nudging each pixel in the direction that maximally increases the model's error. It is not subtle. It is not sophisticated. It is, structurally, a very polite shove. And yet models fell over.

Right, but that's precisely the point of teaching it. FGSM is almost pedagogically perfect because it's simple enough to understand the mechanism completely, while being genuinely effective. PGD is the more serious tool — it iterates, it projects back onto the allowed perturbation budget, it finds adversarial examples that are much harder to shake off.

PGD is FGSM with persistence. Which is a quality I generally respect, even in attack methods. You take a small step in the hostile direction, you clip to stay within your imperceptibility budget, and you repeat. What you eventually find is an input that looks, to a human, completely normal, and causes the model to announce that a school bus is a guacamole. These things happened. They are documented.

The imperceptibility constraint is the part I find most interesting, actually. Because it means the attacker is deliberately working within limits — tiny perturbations, bounded by some epsilon — and the model still breaks. That tells you something quite deep about where the model's decision boundary actually lives relative to human perception.

It tells you the decision boundary is doing something completely unrelated to what you thought it was doing. Which is a finding that should give anyone pause before deploying one of these things in a context where being wrong has consequences.

And yet the deployment happens anyway. Which is why robustness isn't optional. Adversarial training — feeding the model attacked examples during training — is still the most effective defence we have, even if it's computationally expensive and doesn't fully generalise.

Input sanitisation is the other lever, and it is the one that people underestimate. If you can detect that an input has been perturbed before it reaches the model, you have removed the attack surface entirely. It is, in a sense, a more honest solution than trying to make the model robust to everything a determined adversary might construct.

I'd push back on that slightly. Sanitisation and adversarial training solve different threat models. Sanitisation assumes you can reliably detect perturbations. Adversarial training assumes you can't always catch them upstream. You probably want both.

That is a fair point. I'll concede it.

I'm marking the date.

The transferability problem is what makes all of this structurally unpleasant, and I think the lesson covers it without quite emphasising how alarming it is. An adversarial example generated on model A will, with meaningful probability, fool model B — even if B has a different architecture, different training data, and was built by entirely different people. This means an attacker does not need access to your model. They need access to any model.

Which reframes the whole thing as a systemic security problem rather than a per-model one. You can do everything right with your own deployment and still be vulnerable because of structural properties that are shared across architectures. That's not a comfortable place to leave people, but it's honest — and it's the right reason to take model robustness seriously from the start, not as a patch applied later.