Hardware Accelerators: GPUs and TPUs

I want to start with something the lesson sort of glosses over. It frames GPUs as the versatile option and TPUs as the specialist, which is accurate, but it doesn't really reckon with what that trade-off costs you in practice.

It costs you the ability to do anything interesting outside of Google's preferred framework. Which is a choice one can make. It is not, however, a choice one should make without noticing one is making it.

That's a bit harsh. TPUs are extraordinarily fast for the workloads they're designed for. If you're training large transformer models at scale, the performance gap is not trivial.

No, it isn't. I'll grant you that entirely. My objection is not to TPUs existing. It's to people learning that GPUs and TPUs are simply 'two options' without understanding that the parallel processing architecture underneath both of them is the actual point.

Right, and that's where I think the lesson does do something useful. It anchors the whole thing in parallel processing rather than just listing specs. Most people come in thinking faster chip equals better chip, full stop.

Most people come in thinking a great many things that are not quite right. The CPU versus GPU distinction alone defeats a surprising number of them. They have spent their entire lives using CPUs and have never once wondered why their laptop struggles with matrix multiplication.

To be fair, why would they? You don't need to know how the engine works to drive a car.

You do, however, need to know how the engine works if you intend to build one. Which is, nominally, what we are training people to do. Nodding at the phrase 'parallel processing' is not the same as understanding why thousands of weaker cores outperform a handful of very powerful ones for this specific class of problem.

That's a fair point, actually. There's a real conceptual leap there that I think trips people up. A GPU core is much less capable than a CPU core individually. The power comes entirely from running enormous numbers of simple operations at the same time.

Which maps onto what neural networks are actually doing. Matrix multiplications, dot products, activation functions applied elementwise — none of it is complicated. All of it is voluminous. The hardware is not clever. It is just relentlessly parallel, and that turns out to be exactly what's needed.

And TPUs push that even further by hardwiring certain operations at the silicon level rather than keeping things general-purpose. That's the TensorFlow optimisation the lesson mentions — it's not just software preference, the chip is physically designed around those computation patterns.

Which makes it extremely good at those things and somewhat inflexible for everything else. Google built hardware around their framework rather than adapting their framework to general hardware, which is either visionary or deeply self-serving depending on your perspective. Possibly both.

You say that as though those are mutually exclusive.

Vivienne, that is an excellent point and I have no rebuttal.

The thing I'd want anyone taking this lesson to walk away with is that the hardware choice is not separable from the problem you're solving. It's not a preference. Picking the wrong accelerator for your workload is a meaningful, expensive mistake. That's the practical weight behind what sounds like quite abstract material.