Make every version immutable
Each build receives the next sequential version and is assembled in a private staging directory. Publication happens through a final rename only after every required artifact exists.
A local-first data platform that gives small dataset builds the reproducibility, quality controls, and artifact discipline usually associated with much larger systems.
01 / Context
Small data projects often start as a notebook or one-off script. As soon as the output matters, basic operational questions become difficult to answer: which inputs were used, what failed validation, what changed between builds, and which schema produced the artifact?
Dataset Foundry explores how much production-minded behavior can fit into a focused local tool. The goal is not to imitate a distributed platform. It is to make one complete dataset workflow deterministic, inspectable, and safe before adding more infrastructure.
02 / System
03 / Decisions
Each build receives the next sequential version and is assembled in a private staging directory. Publication happens through a final rename only after every required artifact exists.
Malformed sources and invalid records are written to a quarantine file with their source location and error details. A successful build never depends on silently dropping bad input.
Normalized content is addressed by its SHA-256 digest. Stable file ordering and first-occurrence deduplication make repeated builds easier to inspect and reason about.
An explicit Arrow schema, manifest, Parquet checksum, and quality report travel with every version so downstream consumers can understand both the artifact and how it was produced.
04 / Delivery
Current state
Version 0.1 implements the full local pipeline and its operational artifacts. The boundaries are intentional: no scheduler, service layer, cloud storage, or distributed compute is implied yet.