Neurons and Activation Functions

I want to defend the biological analogy here. The lesson maps neurons to actual brain cells, weights to synaptic strength — I think that's genuinely useful scaffolding for a beginner.

It is useful right up until the moment someone walks away thinking a neural network is a brain, which is approximately when the damage begins. The analogy is a ladder you're supposed to kick away. Most people hang from it indefinitely.

Or most people use it correctly and you've met a skewed sample.

That is a fair point, actually. Though I'd argue the sample is less skewed than you'd hope.

The weights section is where I think the lesson earns its keep. It's very clear that learning *is* weight adjustment — that's the whole mechanism, and it says so plainly.

It does, yes. The problem is that 'the network adjusts weights during training' sounds like a description of a process, when it is in fact the entirety of what learning means in this context. There is no other magic happening behind it. Students hear that sentence and nod, and then later ask where the understanding is stored, as though there's a feelings compartment somewhere.

The bias explanation is the one I'd actually push back on in the lesson itself. 'Activates even when inputs are zero' is technically correct but it frames bias as an edge case rather than a fundamental degree of freedom.

Finally, yes. Bias is not a footnote for the unusual scenario where your inputs are all zero. It is a constant offset that shifts where the activation function sits relative to the input. Every neuron has one. It matters every single time. The lesson buries this.

Though to be fair, it does say bias 'shifts the activation function' — that's in the takeaways verbatim.

It says the words. Whether it conveys the significance is a separate question, which I am less confident about.

On activation functions — ReLU, sigmoid, tanh — do you think an intermediate lesson should be explaining *why* you'd choose one over another, or is 'they introduce non-linearity' enough for this stage?

Non-linearity is enough as a concept, but the lesson gestures at 'choosing the right activation function can dramatically impact performance' and then does not elaborate even slightly. That is the intellectual equivalent of saying 'the choice of ingredient matters' in a cookery class and then declining to discuss ingredients.

It's an introductory lesson. You can't front-load vanishing gradients and dying ReLU into the first encounter with the topic.

I'm not asking for vanishing gradients. I'm asking for one sentence that gives a student a reason to care which function they use. 'ReLU is computationally simple and works well in deep networks' takes four seconds to say and costs nothing.

That's actually a reasonable ask. Though I notice you've spent this entire conversation critiquing the lesson's framing rather than its accuracy — which perhaps tells us the content is sound and you're arguing about emphasis.

In technical education, emphasis *is* accuracy. Knowing that bias exists and understanding that it's a parameter with real geometric meaning are not the same thing, and only one of them is useful. But yes. The content is broadly sound.