Data Types and Sources

I want to start with something that actually bothers me about how this topic gets taught. Everyone rushes to the three-way split — structured, semi-structured, unstructured — as if the categories are the point. They're not, are they.

They are not, no. The categories are a means of answering a prior question, which is: what can I actually do with this thing. The taxonomy exists to serve a decision. Most people learn the taxonomy and consider the job done.

Which is exactly backwards. You don't need to know what a CSV is called. You need to know why it behaves differently from a folder of MP4 files when you try to query it.

And yet here we are, teaching people what a CSV is called. I accept that there is a sequencing problem. You cannot leap directly to implications if someone does not know the terms. I simply find the terms get remembered and the implications get forgotten, which is roughly the least useful possible outcome.

To be fair, the lesson does frame it around consequences. The schema point, especially — that a poorly designed schema doesn't just inconvenience you, it actively limits what questions you can ask later. That's not obvious to someone new to this.

It is not obvious, and it is also irreversible in a way that people consistently underestimate. You can clean dirty data. You cannot easily retrofit a schema onto a database that has been accumulating records for three years under a different set of assumptions. That particular mistake is load-bearing by the time anyone notices it.

Which raises something the lesson touches on but doesn't quite land — the relationship between source and structure. The idea that where data comes from largely determines what shape it arrives in.

A sensor does not produce a spreadsheet. An API does not, as a rule, hand you a perfectly normalised relational table. These are not arbitrary facts to memorise. They follow from what the source is doing and why. If you understand the source, the structure is largely predictable.

Although that cuts both ways. If you're building a system and you control the source, you have choices about what structure you impose. Most beginners don't realise that's even a decision they're making.

That is a genuinely good point and I have no contemptuous response to it. You are describing the difference between data as something that happens to you versus data as something you design. Most introductory material treats it as the former.

High praise from you.

I said I had no contemptuous response. I did not say it was extraordinary. The semi-structured category is where this matters most, incidentally — JSON, XML, things with implied schema rather than enforced schema. The flexibility feels like a feature until you are six months in and every record has different keys.

And then you've invented all the problems of unstructured data whilst also convincing yourself you had a plan. That's arguably worse than starting with raw text.

Arguably. I would say definitively, but I appreciate that you're being diplomatic. The lesson is correct that semi-structured is where most people come unstuck in practice. It has just enough apparent organisation to feel manageable, and just enough flexibility to become catastrophic.

So what's the one thing someone should actually walk away knowing from this lesson, in your view.

That classification is a precondition, not a conclusion. You classify data so you can decide something — which tool, which storage approach, which schema, which set of questions is even answerable. If you finish this lesson able to label three circles on a diagram but unable to say what any of it changes about how you work, you have learned a vocabulary and nothing else. That is, in my experience, the most common result.