Schema evolution

Schemas are documents, so schema evolution is document editing — but evolution has rules of its own. datadata’s stance is evolution-first: schemas change freely while the app runs, migrations are explicit, and documents that no longer fit are flagged, never dropped.

Versioning

A schema’s version is its document’s sequence number — there is no separate version field. Every accepted edit to sys:schema:<type> advances it, and every document tracks which schema sequence its data conforms to. Every event in a document’s history also records the schema sequence in effect when it was written — which is what keeps historic versions interpretable anywhere.

Because that number counts this folder’s accepted edits to the schema document, it is a folder-local coordinate: the same type can sit at different sequence numbers in two folders depending on when each folder’s schema was last edited. A version number only means something inside its own folder — there is no global schema version to compare across folders.

Migrations are explicit and append-only

Data-shape changes are declared as migrations in the schema document. Three operations exist, deliberately minimal:

rename — move a field to a new name.
remove — delete a field. Removal is never inferred: dropping a field from the schema without a remove migration makes documents still holding it invalid, rather than silently discarding data.
remap — rewrite scalar values (old → new pairs). Collapsing several old values into one is allowed; one-to-many is not. Remaps are pure — the new value depends only on the old one.

The migration list is append-only, enforced by the server: a schema write may extend it but never modify or drop a committed migration, and the server stamps each appended migration with the schema sequence it took effect at — authors don’t control the stamps.

What authors do control is each migration’s key, and the keys’ lexicographic order is the replay order. A new migration’s key must sort after every committed one — the server rejects a key that would land inside the committed range, since that would silently reorder replay for older documents. Keys can be written by hand or generated by tooling; a convention like 002-rename-title sorts correctly and stays readable.

Migration runs on read — and writes back

When a document is read whose conformed sequence is behind the schema, the server brings it forward: it replays the migrations stamped after that sequence — in their declared order — validates the result (backfilling declared defaults for added fields), and — if anything changed — persists the migrated data back as a normal change with a new sequence number. Each document pays the migration cost once, on its next read, not on every read. There is no proactive bulk sweep: a document nobody reads keeps its old shape, and its pending migrations simply accumulate until it’s next loaded.

Presence schemas don’t migrate

The one exception is a presence schema (sys:schema:presence:<presenceType>). Presence data is ephemeral — never stored, so never brought forward — which means a migration could never fire. So a presence schema carries no migration log at all: a write that declares migrations is rejected. The field shape still evolves in any way you like (add, remove, retype, restructure) purely by editing the schema, since there is no stored presence for the change to strand; live cells simply re-validate against the new shape on their next write.

Invalid documents are flagged, not dropped

Schema edits are not checked against existing documents — you can tighten a type or add a required field freely, and documents that no longer fit become invalid. What happens then is asymmetric on purpose:

Writes are strict. A change that would leave a document invalid under the current schema is rejected (a schemaValidation error on the wire).
Reads are relaxed. An invalid document is still delivered — flagged with the violation, its data intact, its conformed sequence deliberately left behind as the signal. The app (or an agent) decides how to repair it.

Invalidity is discovered on read — nothing sweeps the folder at schema-write time — so a document nobody has loaded since the tightening isn’t known to be broken yet. The discovered invalid documents are enumerable, so “what broke when we tightened the schema?” is a query over what reads have surfaced so far. When you need the full audit, a budgeted validation sweep forces that discovery: it reads every document whose conformed sequence trails the current schema (in host-sized batches, off the hot path), migrating the ones it can and flagging the rest — after which the enumeration is complete, and “is the migration done?” is answerable. Each flagged document carries its violations as located paths, attributed to the schema shape or to a broken cross-document reference.

Unknown fields

Object types and discriminated unions choose how to treat keys the schema doesn’t declare:

reject — the default. An unknown field is a validation issue.
strip — accept the value but drop unknown fields from the validated output, for open-by-design shapes.
keep — accept and retain unknown fields, as long as the retained values are still JSON-compatible.