Where datadata comes from

datadata is a new take, but not from a blank page. It draws on two systems I’ve spent years building — Dossier, an open-source headless CMS, and Shiftic, an AI platform for designing and measuring lasting organizational change. Most of datadata’s design decisions are answers to something one or the other taught me.

Dossier (2020–)

From October 2020 I built Dossier as a solo project — an open-source headless CMS where you bring your own auth, database, and backend, with a schema-driven admin UI on top. It never gained traction, but it’s where most of datadata’s instincts were formed: one idea that survived intact, and several that became problems datadata had to solve.

The complete event stream — the idea that survived. Dossier captured every mutation, schema changes included, as one ordered stream of sync events; replaying it on an empty database reproduced everything, schema and content together, across backends. That idea came through and deepened: it’s datadata’s one timeline principle and the basis of portable event streams.

Three bets aged badly, and each became a problem the build below had to answer:

• Betting on the generic admin UI. A schema-driven admin interface was a pillar of the CMS value proposition — and a large share of the code. AI agents eroded that pillar mid-project: bespoke, task-specific UIs became cheap to build, often cheaper than adapting a generic one.
• Owning a rich text format. Dossier’s rich text format changed twice (Editor.js blocks, then Lexical’s editor state), and each switch meant migrating stored content and rebuilding editor UI.
• A migration rule matrix. Schema changes were governed by a matrix of allowed and unsupported operations, some triggering re-validation or re-indexing duties — expressive, but heavy and easy to get wrong.

One more lesson was structural rather than a pivot. Dossier’s database adapter abstracted over low-level primitives like transactions, across Postgres, SQLite, and D1 — and D1, which has no interactive transactions, proved that boundary wrong rather than just awkward. datadata draws its two extension points higher up: a storage adapter and event bus that own whole operations end to end, with deliberately one production backend until they have earned a second.

Shiftic (2023–)

Dossier is the longer arc, but it isn’t the only system feeding datadata. Since 2023 I’ve been CTO of Shiftic, where we built a comparable engine to power structured UI and AI agents. Shiftic settled on the same core move — JSON Patch as the unit of change, event-sourced — which is evidence, not coincidence, for datadata’s changes-as-data bet.

It also brought the half Dossier never had: realtime sync. Shiftic needed it for the obvious reason — multiplayer humans editing together — and then for a less obvious one: the moment you add AI agents, the UI is multiplayer by default. An agent and a person working the same document are just two participants, so a sync engine stops being a collaboration nicety and becomes the substrate agents run on.

Shiftic had optimistic updates too, but bespoke — hand-rolled for the use cases that needed them rather than a property of the engine, which is the generalization datadata took on.

But Shiftic keeps schema in source code, so data, schema and migrations still live on separate timelines — the exact split that datadata’s one timeline principle and schemas as documents set out to close. Both a positive influence and a working example of the pain it answers.

How datadata came together (2025–)

datadata didn’t start as a library. It started app-first — real applications built directly on Yjs — and the library grew out of what those apps kept reaching for.

In August 2025 that became its own thing: the datadata library, with JSON Patch as the unit of change. Working in it, I immediately missed two things from Shiftic — the JSON-based programming model and having schemas — so schemas went in from the start, defined with Zod.

By October 2025 the cost of JSON Patch for rich text was clear — the same rich-text pain Dossier had already taught, now from the patch side. I explored several alternatives and landed on a hybrid: JSON for structure, Yjs for rich-text fields, so the format question belongs to a project whose whole job it is. More demo apps followed as stress tests, and the library refined and settled.

In May 2026 the Zod schemas gave way to datadata’s own declarative schemas, which also made it possible to change a schema at runtime. In place of Dossier’s rule matrix, the answer is deliberately smaller: three explicit, append-only migration operations, with documents that no longer fit flagged rather than blocked or dropped.

In June 2026 came staged sessions: a way to see and approve changes to a document before they land — an AI agent’s work that a person signs off on, or human edits held back for review the same way. Dossier had a coarse precursor — a published/draft split on every document — but staged sessions are finer-grained. Promoted to a library primitive, they’re the review surface that replaces Dossier’s generic admin UI, now that the API is small enough for an agent’s tool definitions. The same release reworked how Yjs content is stored and synced, and added a presence system.

In July 2026 two kinds of hardening landed. Offline support became real: an opt-in persistence adapter carries both writes and reads across going offline — queued writes replay on reconnect, cached documents render immediately, and a reload is just a slow reconnect. It stays a property of sync and optimistic updates rather than a separate mode. And authorization moved into the library: one principal model whether the writer is a person’s frontend, trusted server code, or an AI agent, with declarative access rules that live in the schema and per-document read filtering. Dossier left auth entirely to the host app; agents as ordinary participants — the multiplayer-by-default point above — are what pushed access control into the substrate: an agent should get a narrower grant than the person it works alongside.

The thread

The thread runs through both. Dossier was content infrastructure with sync underneath; Shiftic put structured UI and AI agents on a realtime engine but kept schema in code. datadata promotes what each got right to the product itself — Dossier’s self-contained, schema-aware event log and Shiftic’s changes-as-data over realtime sync — and sheds the layers that aged worst.