What are schemas for and do we really need them?
The word “schema” comes up pretty often in discussions about data; either directly, or in the form of its near-antonym adjective form “schemaless”.
A schema is a design, or maybe a specification, for the shape of data; perhaps it describes how the data is stored physically, like the schema of a SQL database, or perhaps it defines what is and isn’t valid data, like an XML or JSON schema.
But describing what a schema is will only be a by-product of this article - our focus is on why we make schemas.
Are schemas compulsory?
Some systems require a schema to store data at all. In a SQL database, one needs to explicitly create tables and define what columns they have and the types of those columns, because the database needs this information to know how to represent the records in the computer; it needs to know how large a record can be, and to decide where to store the fields of the records so it can find them again.
But some data storage systems don’t require schemas, because they store the records in a form more like JSON; this comes at a cost in efficiency, but brings flexibility - different records may have different structures, perhaps better representing the complexity of real data, and data can be stored without deciding on its structure in advance.
And some occupy middle grounds - such as SQLite, which requires you to list the columns in a table and give them names - but types are optional, and even if you DO specify column types, you’re still free to put data in a column that doesn’t match the column’s type.
Data processing and storage tools you use might require a schema, but even when you’re using ones that don’t, there will almost always be a schema somewhere even if the tool is unaware of it. Usually, the records will follow some structure, even if it’s flexible; there will be some rules as to what’s expected and what isn’t, even if they’re unwritten. But even systems with compulsory, rigid, schemas often only know about the parts of the schema required to store the data. A field might be declared as “string”, but the system relies on it being a valid postcode. A field might be declared as “optional”, but we rely on there never being a case where a latitude is given but not a longitude (or vice versa). The schemas needed for tools to know how to represent data aren’t expressive enough to say these kinds of things, because the tools don’t need that information.
These kinds of “schemas for tools to know how to represent data” are what first comes to mind for data engineers who grew up on SQL databases and programming-language struct/record types, but they’re a distraction from the real benefit of schemas.
Schemas as expectations
In a data processing system, there will always be expectations about the shape of data. Even if you’re using a schemaless database, your application that pulls records from it will probably assume certain fields are present, and have data of certain types, and will be unable to process the records if they don’t. Even if the software just displays the record to a human user, and will do that for any data it finds, the human will probably have expectations; if a user pulls up a purchase record and it doesn’t say what was bought, who bought it, what the price was, and when it happened, they’ll be pretty disappointed.
If a schema is unwritten, represented only as expectations in various bits of software throughout your system, that can be fine - but you run an increasing risk that the expectations embedded in different places don’t completely match up. So it’s often wise to define it somewhere, so that developers and users can know what they should expect. This can be as simple as prose, in documentation or comments somewhere, but if it’s written in some kind of “schema language” it becomes possible to automatically check that data matches the schema. This is useful in automated test suites, and in setting up checkpoints where data entering the system from external sources can be checked for validity once and rejected if it’s erroneous, saving the system from re-checking validity subsequently and allowing developers to assume data is valid within the system’s boundaries.
But any schema language will have limits. There will be useful properties we’d like to enforce that can’t be written in any given language, due to it lacking the features to express that. And there are fundamental limits to what a computer can decide - such as Godel’s incompleteness theorem and the halting problem; no amount of adding features to a schema language will be able to overcome those. And maybe a field in the data has to contain an identifier for some data contained in another database; the schema checker may not have access to that database because of architectural or security constraints in the system, or that data might be sent later so we can’t verify it now, or access to that database might be slow and expensive. So any formal schema language may still require additional notes to be written in prose, to capture constraints that can’t be expressed in the language.
Schemas for automation
Another reason to express the structure of data in a machine-readable form is to allow for automation. Checking for validity is really just one example of this, but there’s a lot more that can be done.
One common case is to automate conversions between data representations. ETL tools will often use schema information to automate conversions, and tools like Swagger use a schema definition to automatically generate code to convert between network message formats and data structures in memory for a given programming language. This saves programmers a lot of tedious work!
But schemas can automate data tools for non-programmers, too, if they contain more problem-domain information than merely representational “this field is a string” information. If a database knows that some fields contain postcodes specifically, rather than just that they’re strings (or even “strings that match a certain pattern that looks like a postcode”), then when viewing data with postcodes in a browsing tool can automatically link those postcodes to records in other tables that mention the same postcode. This doesn’t require the system to have a hardcoded notion of what a postcode is - which would of course not exist for some application-specific ID anyway – it merely requires the schema language to be able to define a postcode type once and reference it in different parts of the system.
We might even be able to express the fact that postcode values can be sent to Google Maps to be shown on a map, or that your internal order IDs can be embedded in links to your internal order processing system to see their details, by annotating the postcode or order ID types with a URL template that values of that type can be dropped into to make links - and then a user interface showing data can enrich its interface without any custom programming.
Data processing, too often, requires one to hire specialists to build custom tools; part of the reason for this is that schemas are often unwritten, or written in technical prose, and so it’s impossible for a tool to “understand” the data well enough to process it. A human has to figure it out and do the task, or build a bespoke tool that non-specialists can use to do it. But richer schema languages that express more information about the meaning of the data gives more scope for generic tools, usable by non-specialists, to do more useful things.
Conclusion
While most people think of something like a SQL database layout, or maybe a JSON schema, when the word “schema” comes up; those are just the most basic forms of schema. Even “schemaless” systems have a schema, but it might be hard to find because it’s a set of assumptions scattered around different parts of a system – which can make it hard to understand and evolve a system (or to be confident it’s not full of edge-case bugs!)
Schemas can simply express what fields you expect in what kinds of records, and whether they’re numbers or text or yes/no choices, and whether they’re optional; but can include subtler information, such as that it’s a name or a postcode or a price in pounds rather than just “string” or “number”. These problem-domain schemas allow generic tools to automate more and more complex tasks, reducing the tedium and expense of specialist engineers having to do things and build custom tooling.
If you’d like help understanding the structure of your data, come and talk to the experts!
Author
Tags: