CLI conventions

Use the pencil to edit title, status, priority, and owner. Changing status auto-prepends a changelog entry.

Pin the cross-cutting CLI conventions before per-verb specs land. Today
the workspace ships two per-service Typer CLIs (josh-ingester,
josh-embedder) that operators invoke via kamal app exec --reuse '...',
and three planned specs (cli-init, cli-ingest,
embedding-snapshot-distribution) reference a unified josh binary
that doesn't yet exist. Without this layer, each spec invents its own
verb shape, output format, and exit-code mapping, and the OSS first-run
UX fragments across three CLI names.

This spec is decisions-as-acceptance-criteria. Implementation lives in
a new josh-cli/ workspace member that imports the per-service
packages in-process (or wraps kamal app exec for remote dispatch).
Per-service CLIs stay as in-container CMDs and as the in-process
targets the umbrella imports — they don't go away.

1. Where Josh is installed with the CLI extra (`pip install josh-foundation[cli]`), the binary `josh` shall be available on PATH and shall expose a Typer-based command tree.
2. When `josh init` is invoked for the first time, the system shall walk the user through deployment-mode selection (`local`, `kamal`, `compose`, `cloud`) and write `~/.josh/config.toml` containing the chosen mode plus connection details.
3. When any command runs in `mode = local`, the system shall execute the per-service Python (e.g., `ingester.runner.run_source`) in-process against the configured substrate path.
4. When any mutating command runs in `mode = kamal`, the system shall dispatch via `kamal app exec --reuse '<josh-equivalent> ...'` against the configured `deploy_dir` and `target_service`.
5. Where the active context's `mode` is `compose` or `cloud`, mutating commands shall exit with code 4 and a message that the mode is reserved but not implemented in v1.
6. When `josh --help` is shown, the help output shall list top-level commands (`init`, `status`, `ingest`, `backfill`, `search`, `migrate`, `logs`) separately from component groups (`embed`, `schedule`, `source`).
7. Where a verb is high-frequency, single-purpose, and OSS-first-hour-relevant, it shall live at the top level (`josh ingest <source>`, not `josh source ingest <name>`).
8. Where a component has three or more sub-verbs (`embed daemon`/`drain`/`status`; `schedule list`/`run`; `source list`/`checkpoint`/`clear`), it shall be exposed as a grouped subcommand.
9. When `josh init` runs with `mode = local` and the user accepts the default, the system shall create `./data/josh.db` relative to the cwd, run `alembic upgrade head` against it, and write `./josh.toml` referencing the path.
10. When any command resolves the substrate path, the priority order shall be: (1) `--db PATH` flag, (2) `JOSH_DB_PATH` env var, (3) `./josh.toml` `[default].substrate_db_path`, (4) `~/.josh/config.toml` active context's `substrate_db_path`, (5) error with `code='no_substrate'` and a hint to run `josh init`.
11. Where `josh init` is run against an existing config, the system shall print 'config already exists, use --force to overwrite' and exit 0; with `--force`, the existing config shall be replaced.
12. When the substrate path resolves but the file does not exist, the system shall create it (with parent directories) and run migrations before the first read; subsequent invocations shall not migrate unconditionally.
13. When `--json` is passed to any command, the system shall emit only structured JSON to stdout; logs and progress shall go to stderr.
14. When `--json` is absent, the system shall emit human-readable output (tables, progress bars, color when output is a TTY).
15. Where output is not a TTY (pipe, redirect, container CMD), color shall be disabled and progress bars suppressed; the format shall not auto-switch to JSON — `--json` is always explicit.
16. When `NO_COLOR=1` is set in the environment, color shall be disabled regardless of TTY status.
17. When a long-running command (`ingest`, `backfill`, `embed daemon`, `migrate`) is invoked with `--json`, output shall be NDJSON — one JSON object per line, each with a top-level `type` field; the final line shall always be `type: 'result'` carrying the run summary.
18. When a one-shot command (`status`, `search`, `schedule list`, `source list`, `checkpoint`) is invoked with `--json`, output shall be a single JSON object on stdout.
19. Where any command emits data on stdout, it shall not interleave logs/progress on stdout — those go to stderr — so `josh ingest <source> --json | jq` parses cleanly.
20. When a command completes successfully (including empty-result-set cases such as `josh search` with zero matches), the exit code shall be 0.
21. When a command fails generically (unhandled exception, config error, ingest run with all records errored and zero successes), the exit code shall be 1.
22. When `josh ingest` produces partial results (some records succeeded AND some errored), the exit code shall be 2 and the JSON `result` line shall include the per-bucket counts.
23. When a per-source advisory lock is held by another process, the exit code shall be 3 and the message shall identify the lock holder if available.
24. When CLI input is malformed (unknown source name, invalid ID format per rest-api-conventions, bad date), the exit code shall be 4 and the structured error response shall include a `hint` field with `format` / `example` / `valid_values` per the conventions.
25. When the process receives SIGINT, the exit code shall be 130; SIGTERM shall be 143 (POSIX `128 + signal_number` convention).
26. When `--help` is shown for any command, the footer shall reference the exit code table; `josh exit-codes` shall print the full table on demand.

None.

• substrate-sqlite-foundation
• substrate-migrations-workflow
• rest-api-conventions

## Architecture

Three layers:

1. josh umbrella (new josh-cli/ workspace member). Thin Typer
dispatcher. Reads ~/.josh/config.toml (and per-project ./josh.toml),
determines the active deployment mode, and dispatches each command
either in-process (imports ingester.runner, etc.) or as a subprocess
wrapping kamal app exec. Provides one binary on PATH for humans.

2. Per-service CLIs (josh-ingester, josh-embedder, existing).
Stay as today. Used as:
- In-container CMDs (josh-ingester daemon, josh-embedder daemon).
- In-process import targets for mode = local.
- Wrapped commands for mode = kamal (the umbrella shells out to
kamal app exec --reuse 'josh-ingester run X').

3. Substrate / API surfaces (josh-core REST, josh_substrate package).
Unaffected; the CLI is a humans-only client.

## josh-cli/ package layout

``
josh-cli/
├── pyproject.toml
├── README.md
└── src/
└── josh_cli/
├── __init__.py
├── cli.py # Typer app, top-level entry
├── config.py # ~/.josh/config.toml + ./josh.toml loading
├── modes/
│ ├── __init__.py
│ ├── local.py # in-process dispatch
│ ├── kamal.py # kamal app exec wrapper
│ ├── compose.py # reserved (raises NotImplementedError in v1)
│ └── cloud.py # reserved
└── commands/
├── init.py # josh init interactive walkthrough
├── status.py
├── ingest.py
├── backfill.py
├── search.py
├── migrate.py
├── logs.py
├── embed.py # josh embed daemon | drain | status
├── schedule.py # josh schedule list | run
└── source.py # josh source list | checkpoint | clear
``

Distributed via two install groups in the root pyproject.toml:

- pip install josh-foundation — substrate + per-service workers, no
josh binary. For container images.
- pip install josh-foundation[cli] — adds josh-cli and the josh
entry point. For human installs.

## Mode detection

josh init writes ~/.josh/config.toml:

```toml
default_context = "main"

[contexts.main]
mode = "local" # or kamal | compose | cloud
substrate_db_path = "./data/josh.db"

# mode = kamal:
# deploy_dir = "/Users/ritz/projects/josh/josh-core"
# target_service = "josh-ingester"

# mode = cloud:
# api_url = "https://api.josh.dev"
# api_key_env = "JOSH_API_KEY"
```

Per-project shadow at ./josh.toml:

``toml
[default]
substrate_db_path = "/Users/ritz/projects/foo/data/josh.db"
``

Active context resolves by:
1. --context NAME flag (per-invocation).
2. JOSH_CONTEXT env var (per-shell).
3. default_context in ~/.josh/config.toml.

Substrate path resolves by:
1. --db PATH flag.
2. JOSH_DB_PATH env var.
3. ./josh.toml [default].substrate_db_path.
4. Active context's substrate_db_path.
5. Error.

## Verb taxonomy

Top-level (flat) — single-verb operations the OSS first-time user
invokes in the first hour:

| Verb | What it does |
|------------|--------------|
| init | First-run walkthrough; writes config, creates substrate, runs migrations. |
| status | Aggregate health: substrate size, worker state, last cron fires per source. |
| ingest | josh ingest <source> [--since DATE] — incremental run. |
| backfill | josh backfill <source> — full re-pull (no watermark). |
| search | josh search "query" [--source X,Y] [--since DATE] — substrate query. |
| migrate | alembic upgrade head. |
| logs | josh logs <source> [-n N] — tail recent ingestion_logs rows. |

Grouped — components with three+ sub-verbs:

| Group | Sub-verbs | Notes |
|-------|-----------|-------|
| embed | daemon, drain, status | Embedder worker operations. Maps to josh-embedder per-service CLI. |
| schedule | list, run | Scheduler operations. list shows next-fire times. |
| source | list, checkpoint <name>, clear <name> | Per-source operator triage. clear is destructive (gated by --force). |

Adding a new verb: top-level if it's a single-verb high-frequency
user-facing operation; grouped if it's a third sub-verb under an
existing component. New components (a fourth group) are rare — the
three above cover the v1 surface.

## Output format

Default = human, --json for machine. NDJSON for streams (typed lines
with type discriminator), single object for one-shots. stdout = data,
stderr = logs. Color/progress only when TTY + not NO_COLOR. Format
never auto-switches.

Streaming result line schema (every long-running command):

``json
{
"type": "result",
"command": "ingest",
"exit_code": 0,
"summary": { ... command-specific stats ... },
"duration_seconds": 142.7,
"completed_at": "2026-05-10T20:02:22.123Z"
}
``

## Exit codes

| Code | Meaning |
|------|---------|
| 0 | Success (incl. empty-result success). |
| 1 | Generic error / all-failed. |
| 2 | Partial success (some records OK, some errored). |
| 3 | Lock held (per-source flock contention). |
| 4 | Invalid input (bad source name, malformed ID, validation). |
| 130 | SIGINT (user Ctrl-C). |
| 143 | SIGTERM (orchestrator kill). |

Documented inline in every command's --help footer; full table at
josh exit-codes.

## Implementation phasing

- Phase 1: scaffold josh-cli/, write josh init walkthrough,
implement josh ingest / josh backfill for mode = local only.
- Phase 2: mode = kamal dispatch via subprocess.run(["kamal", "app",
"exec", ...]).
- Phase 3: josh embed, josh schedule, josh source groups.
- Phase 4: josh search, josh status, josh logs.
- Phase 5: --json NDJSON streams + exit-code wiring.
- Phase 6: contract tests covering every acceptance criterion;
success_determiner flips green.

1 of 12 done.

• t1 All 5 cross-cutting decisions resolved with rritz; spec drafted
• t2 josh-cli/ workspace member scaffolded with package layout above
• t3 Config loader (~/.josh/config.toml + ./josh.toml) with active-context resolution
• t4 Mode dispatch: local (in-process) and kamal (subprocess wrap kamal app exec)
• t5 josh init walkthrough writes config interactively; --force overrides
• t6 Top-level commands: init, status, ingest, backfill, search, migrate, logs
• t7 Grouped commands: embed (daemon/drain/status), schedule (list/run), source (list/checkpoint/clear)
• t8 --json output mode: NDJSON for streams, single object for one-shots, stderr for logs
• t9 Exit codes wired through every command; josh exit-codes prints the table
• t10 josh-cli/tests/test_cli_conventions.py covers every acceptance criterion
• t11 cli-init + cli-ingest + embedding-snapshot-distribution declare dependency on this spec
• t12 Determiner runs green — all conventions enforced via contract tests

• 2026-05-10T21:00:00Z (new)→planned Spec authored after a 5-question grilling-style decision pass with claude. Decisions baked into acceptance criteria; implementation deferred to a new josh-cli/ workspace member that will land in the coming weeks. Three downstream specs (cli-init, cli-ingest, embedding-snapshot-distribution) will declare this spec as a dependency in the same change set so they inherit the umbrella taxonomy + verb conventions instead of inventing per-spec shapes. One operational note: the per-service CLIs (`josh-ingester`, `josh-embedder`) stay as today — they're still the in-container CMDs and the in-process import targets the umbrella uses. The umbrella is additive, not a replacement.