Spec workflow

How to read, edit, add, and verify spec items. docs/spec/index.html is the source of truth for what Josh Foundation v1 ships; this page is the reference for working with it.

Anatomy of a spec

Each spec item is a YAML file at docs/spec/data/<id>.yaml. The renderer (bin/build-spec.py) and the in-browser editor both consume the same JSON Schema at docs/spec/data/_schema.json.

Required fields:

Field	Purpose
`id`	kebab-case, must equal the filename stem.
`title`	Sentence-case, no trailing period.
`category`	`substrate` (cross-cutting machinery), `source` (per-data-source ingester), `surface` (REST/MCP/CLI/UI), or `launch` (OSS launch artifacts).
`status`	See lifecycle.
`why`	1-3 sentences. The deliverable + intent. Anthropic's framing: "ambitious deliverables, not steps."
`acceptance_criteria`	EARS-form sentences (When …, While …, Where …, If … then …, or ubiquitous "The system shall …").
`success_determiner`	Tagged union — load-bearing. An agent runs this to decide if the spec is met. Never free prose.

Optional but encouraged: user_stories, priority (p0/p1/p2), dependencies, plan, tasks, out_of_scope, clarifications_needed. changelog is append-only — auto-prepended when the in-browser editor flips status; if you edit YAML directly you must append entries by hand.

Success determiner kinds

Pick the most mechanical kind that fits. manual is a fallback, not a goal.

Kind	Use when	Required fields
`bash`	A shell command can prove the spec is met.	`command`, `expect`
`sql`	The proof is data in the substrate DB.	`db`, `query`, `expect`
`test_file`	A test file in the repo encodes the contract.	`path`, `runner`
`expected_output`	The proof is a string in command stdout, file contents, or a URL response.	`source`, `expect`
`manual`	Only when no mechanical check is possible. Lists items a human (or computer-use agent) confirms.	`checklist`

Status lifecycle

draft → planned → in_progress → blocked → verified → shipped

draft — idea, not committed.
planned — scoped, not started. Acceptance criteria and a real success determiner exist.
in_progress — active work.
blocked — waiting on a dependency.
verified — success determiner ran green.
shipped — change is live in production. Always reached through verified, never directly.

verified ≠ shipped. Verified means the determiner passes locally or in CI; shipped means the change is live and inventory pages reflect it.

Workflow rules — apply before every status flip

Before planned → in_progress: dry-run the determiner with bin/dry-run-spec.py <id> --explain. It must fail because the implementation doesn't exist yet — not because the determiner is malformed. If it fails for typos or missing files, the spec itself is buggy. Fix it before flipping.
Before in_progress → verified: run bin/build-spec.py --lint, run the determiner (must pass), and append a changelog entry. The build script does not auto-prepend; only the in-browser editor does.
Before verified → shipped: confirm the change is live and both inventory pages — josh-data-sources.html and data-status.html — reflect it in the same commit. Bump Last updated on data-status.
After shipping a substrate-level architectural decision: grep all draft/planned/in_progress specs for references to the old pattern. Update or invalidate them in the same change.
When you find yourself adding a special case to a doc ("this source is different because…"), pause and consider factoring the code instead. The third instance of a pattern is the inflection point.

Adding a new spec

Per-source ingester specs use the source template, which bakes in the 14-task ingestion-built contract:

cp docs/spec/data/_templates/source.yaml docs/spec/data/<your-source>.yaml
# edit the yaml — at minimum: id, title, why, acceptance_criteria, success_determiner

bin/build-spec.py        # regenerate spec HTML
bin/sync-nav.py          # propagate the canonical sidebar to peers

For non-source specs (substrate, surface, launch), copy any existing seeded example as a starting point. Keep the ID kebab-case and matching the filename stem — the build script will reject the file otherwise.

Editing a spec

Two paths:

In-browser editor. Open docs/spec/index.html locally (via ./serve.sh) and click any pencil icon. Chrome, Edge, and Arc write directly back to the YAML via the File System Access API. The editor auto-prepends a changelog entry when status changes and auto-bumps updated_at.
Direct YAML edit. Edit the file in docs/spec/data/, then run bin/build-spec.py to regenerate HTML. You must append the changelog entry by hand if status changed.

Hygiene tools

Command	What it does
`bin/build-spec.py`	Regenerates HTML from YAML. Validates every spec against the JSON Schema first; refuses to write on a schema violation.
`bin/build-spec.py --lint`	Heuristic grounding pass. Warns on stub-drift, stale `[STUB]` placeholders left in non-draft specs, and backtick-quoted identifiers (file paths, function names) that don't exist in the implementation corpus.
`bin/build-spec.py --lint --strict`	Same, but lint findings exit nonzero. Wire this into CI.
`bin/dry-run-spec.py <id>`	Runs the spec's `success_determiner` against current repo state. Returns pass/fail.
`bin/dry-run-spec.py <id> --explain`	Same, plus prints what the determiner ran and what it expected.
`bin/sync-nav.py`	Propagates the canonical sidebar from `docs/index.html` to every peer page. Run after adding or renaming any nav entry.
`bin/sync-nav.py --check`	CI-friendly variant: nonzero exit if any peer has drifted from the canonical.

When to mark a spec out of scope

If a planned spec turns out not to be needed, prefer deletion over a perpetual draft. A directory of unstarted ideas dilutes the spec system's signal — the spec page should describe what Josh ships, not what someone once wrote down. Archive the rationale separately if it's worth keeping.