For business
What it is
Putting everything into one consistent shape: the same field names, date formats and record structure, whatever the source.
Why it matters
One shape means one set of rules downstream, cheaper to build and easier to audit, and new sources plug in without rewiring the pipeline.
Risk without it
Every later step grows special cases per source, and bugs hide in the differences.
For engineers
How it works
A canonical document schema (source, type, UTC timestamps, language, speakers, text, provenance) is defined once, and each source has a mapper that outputs it. Provenance links every normalised record to its raw object.
Typical stack
- TypeScript
- Zod / JSON Schema
- Lambda or container workers
Pitfalls & trade-offs
- Version the schema: old records will need migrating.
- Normalise time zones at the edge, never later.
- Keep the original text next to the cleaned text for audits.
In this example
In
→ transcript.txt fact-find.pdf → text
Out
{ source: "meeting",
speakers: 2,
date: 2026-10-12 } Fictional sample · not a client system