REST API conventions

Use the pencil to edit title, status, priority, and owner. Changing status auto-prepends a changelog entry.

Pin the cross-cutting REST conventions every endpoint, every resource,
and every consumer (REST clients, MCP server, CLI) inherits from. Without
this layer, each downstream spec (rest-api-resource-endpoints,
rest-api-search, mcp-server) silently picks its own pagination shape,
error format, datetime style, and field-projection model — and when
agents trained on different APIs hit Josh, they drift.

This spec is decisions-as-acceptance-criteria. Implementation lives in
josh-core's FastAPI app + a shared error/pagination/serialization
module under shared/josh_substrate/. Contract tests pin the shape so
drift is caught at CI time, not in production.

1. When a list endpoint is called, the response shall return at most `limit` records (default 50, max 200), and shall include `next_cursor` (string or null) and `has_more` (boolean) fields in the JSON body — never in the `Link` header.
2. When a list endpoint is called with `?cursor=<token>`, the response shall return the next page starting from that opaque cursor; cursors shall be encoded as base64 of `{sort_key, id}` so they survive across writes.
3. When the search endpoint is called, pagination shall use `?offset=<n>&limit=<n>` (offset default 0, limit default 50, max 200), and the response shall include `total` (integer), `limit`, and `offset` fields.
4. Where a list endpoint is called with `?include_total=1`, the response shall additionally include a `total` field; otherwise `total` is omitted.
5. When a list or search endpoint returns successfully, the response body shall be a JSON object whose top-level `data` field is an array of records.
6. When a singleton endpoint returns successfully, the response body shall be the record itself (no `data` wrapper).
7. Where a record has an `id` field, it shall always be included in the response regardless of any `?fields=` projection.
8. When a request fails, the response body shall be `{"error": {"type": ..., "code": ..., "message": ..., "request_id": ...}}` and shall never be a bare string.
9. Where the failure is an `invalid_request` caused by a malformed ID, datetime, or constrained-value field, the `error` object shall include a `hint` field with at minimum `format` and `example`, plus `valid_values` where the input is a finite enum.
10. When a request involves multiple field validation errors, the `error` object shall include a `fields` array of `{path, message}` entries.
11. When any request completes (success or error), the response shall include an `X-Request-Id` header containing a UUIDv7 string; on error responses the same value shall appear in `error.request_id`.
12. Where the API emits an error, the `error.type` shall be one of: `authentication`, `permission`, `not_found`, `invalid_request`, `rate_limit`, `server_error`, `service_unavailable`.
13. Where the API emits an error, the `error.code` shall follow `<resource>_<reason>` snake_case naming (e.g., `bill_not_found`, `key_revoked`, `rate_limit_exceeded`).
14. Where a REST endpoint is exposed publicly, the URL shall begin with the `/v1/` path prefix; `/v0/` is reserved for unreleased/internal endpoints with no compatibility promise.
15. When a breaking change ships, the previous major version shall remain reachable for at least 6 months with a `Sunset` header before returning HTTP 410 Gone.
16. Where a resource has a composite natural key, the public ID shall be a single colon-delimited lowercase string with type-first ordering (e.g., bills: `hr:119:1`; roll-call votes: `house:119:2026:142`; CREC granules: `crec:2026-05-10:S1234`).
17. Where a resource has an already-opaque natural key (`bioguide_id`, CRS `R47892`, FR `2026-08558`), the public ID shall pass through unchanged.
18. Where a sub-resource is exposed (versions, cosponsors, body), the URL shall nest under the canonical opaque ID (`/v1/bills/hr:119:1/cosponsors`, never `/v1/bills/119/hr/1/cosponsors`).
19. When an API response includes a datetime instant, it shall be ISO-8601 / RFC 3339 with UTC offset `Z` and millisecond precision (e.g., `2026-05-10T18:30:00.123Z`).
20. Where a substrate field stores a date-only value (CRS `published_at`, FR `publication_date`, bill `introduced_date`), the API shall return it as `YYYY-MM-DD`; the system shall not promote it to midnight UTC.
21. When an API request includes a datetime parameter, the system shall accept any RFC 3339 string (with `Z`, with offset, without seconds, without milliseconds, or date-only) and normalize to UTC server-side.
22. When a list or search endpoint accepts filter parameters, they shall be expressed as flat query params (`?status=enacted&congress=119`); the API shall not require POST bodies, JSON:API `filter[]` brackets, or operator suffixes for v1 filtering.
23. When a multi-value filter is supported on a field, the API shall accept a comma-separated list (`?source=fr,bills,crs`).
24. When `?since=<datetime>` is passed, results shall be restricted to records where the endpoint's documented primary time field is greater than or equal to the value; `?until=<datetime>` shall be exclusive less-than.
25. Where a request uses a query-param name reserved by these conventions (`q`, `since`, `until`, `cursor`, `limit`, `offset`, `sort`, `fields`, `include_total`, `source`), no resource-specific filter shall shadow it.
26. When `?sort=<field>` is passed, results shall be sorted ascending by `<field>`; `?sort=-<field>` shall sort descending.
27. When `?sort=<key1>,<key2>` is passed, results shall be sorted first by `key1`, then by `key2` as tiebreaker.
28. When the search endpoint is called without `?sort=`, results shall be ordered by hybrid BM25+vec relevance descending.
29. When the search endpoint is called with `?sort=<field>`, the system shall identify a candidate pool of `min(1000, max(200, offset + 2 * limit))` results by hybrid relevance, re-sort that pool by `<field>`, and return the user's offset/limit slice.
30. Where a list endpoint is called without `?sort=`, the default sort shall be `-<primary_time_field>` for that resource (newest first).
31. When a singleton endpoint is called without `?fields=`, the response shall include the documented `full` fieldset for that resource (every metadata field plus body where the resource has one).
32. When a list or search endpoint is called without `?fields=`, the response shall include the documented `card` fieldset for that resource (id, primary date, title, citation_string, source_url, no body).
33. When the search endpoint is called, each card shall include a `snippet: {text, highlights: [{start, end}]}` field with a body excerpt (max 200 chars) from the matched section, with offsets — never raw HTML.
34. When the search endpoint is called, each card shall include a `body: {size_bytes, url}` object pointing to the body endpoint, or `body: null` for resources with no body.
35. When `?fields=<comma-list>` is passed, the response shall include only the listed fields plus `id` (always present); unknown fields shall produce HTTP 400 with `error.code='invalid_field'` and `hint.valid_values` listing valid fields.
36. Where a resource has a body field, a dedicated `/v1/<resource>/<id>/body` endpoint shall be available returning body fields only.
37. When authentication fails, the response shall be HTTP 401 with `error.type='authentication'` and one of: `key_missing`, `key_malformed`, `key_invalid`, `key_revoked`.
38. When the requesting account's plan does not include a feature, the response shall be HTTP 403 with `error.type='permission'` and `error.code='plan_insufficient'`, including the current plan and an upgrade pointer in `error.hint`.
39. When a requested resource has no record in the substrate, the response shall be HTTP 404 with `error.code` of `record_not_found` (for IDs) or `endpoint_not_found` (for unknown URL paths).
40. When a requested resource was previously ingested but has been withdrawn or superseded upstream, the response shall be HTTP 410 Gone with `error.code='record_withdrawn'` or `record_superseded`, and where a successor exists, `error.hint` shall include `successor_id` and `successor_url`.
41. When the rate limit is exceeded, the response shall be HTTP 429 with `error.type='rate_limit'` and `error.code='rate_limit_exceeded'`, plus a `Retry-After` header (seconds).
42. When any response is returned, the response headers shall include `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset` (epoch seconds, UTC).
43. Where a 429 response is returned, the `Retry-After` header shall be present and `error.hint` shall include `retry_after_seconds`, `limit`, and `window` (humanized: `minute` / `hour` / `day`).

None.

None.

Locked decisions, organized by section. Each becomes one or more EARS-form
acceptance criteria above, and each maps to a contract test in the
determiner.

## 1. Pagination

- List endpoints use cursor pagination. Cursor is base64 of {sort_key, id}.
Response carries next_cursor + has_more in the JSON body. total is
opt-in via ?include_total=1 to avoid SQLite COUNT(*) cost on every
request.
- Search endpoint uses offset + limit. Response carries total, limit,
offset. BM25+vec relevance score is per-query so cursor doesn't compose
well; offset is fine for top-K agentic search.
- Default page size 50, max 200. Tuned for agent-shaped consumers (bigger
than human-pagination defaults).
- Pagination metadata in JSON body, never in Link header. Agents parse
JSON; headers are an extra integration surface.

## 2. Response envelope

Stripe-style flat envelope:

- List/search: {data: [...], next_cursor, has_more} or {data: [...], total, limit, offset}.
- Singleton: bare object — no data wrapper. GET /v1/bills/hr:119:1
returns {"id": "hr:119:1", ...} directly.
- Wrapper field name: data (matches Stripe, GitHub, Anthropic, OpenAI).
- Cursor field name: next_cursor + has_more (clear, paired).

## 3. Error envelope

Custom envelope, Stripe / OpenAI / Anthropic flavor:

``
{
"error": {
"type": "<category>", // 7-value enum
"code": "<resource>_<reason>",
"message": "<human-readable>",
"request_id": "req_01HF6...",
"hint": { ... }, // optional structured help
"fields": [ ... ] // optional, multi-field validation
}
}
``

Type enum (load-bearing — agents branch on it):
authentication, permission, not_found, invalid_request,
rate_limit, server_error, service_unavailable.

Code namespace: <resource>_<reason> snake_case. Documented as
additions land. Examples: bill_not_found, key_revoked,
query_too_long, embedding_provider_down, record_withdrawn.

hint shape (where applicable):
- For format errors: {format, example, valid_values?}.
- For rate limits: {retry_after_seconds, limit, window}.
- For permission errors: {current_plan, upgrade_url}.
- For 410 withdrawn/superseded: {successor_id, successor_url, withdrawn_at}.

X-Request-Id: <uuidv7> on every response (success + error). Body echoes
the same value on errors.

## 4. Versioning

/v1/ path prefix on every public endpoint. /v0/ reserved for
unreleased/internal. Minor changes (additive) ship under /v1; breaking
changes cut /v2 with at least 6 months of overlap, Sunset header on
the deprecated version, then HTTP 410.

## 5. IDs

Composite natural keys → colon-delimited lowercase strings, type-first.

- Bills: hr:119:1, s:119:1234, hjres:119:5, hres:119:42.
- Roll-call votes: <chamber>:<congress>:<year>:<vote_number> —
e.g., house:119:2026:142.
- CREC granules: crec:<issue_date>:<granule_id> —
e.g., crec:2026-05-10:S1234.
- Public Laws: pl:<congress>:<number> — e.g., pl:119:1.

Already-opaque natural keys pass through unchanged
(legislators' bioguide_id like S000033, FR 2026-08558,
CRS R47892).

Sub-resources nest under the canonical opaque ID:
/v1/bills/hr:119:1/cosponsors.

Substrate primary-key columns (bill_id TEXT PRIMARY KEY) carry the
same string form so citations can quote them directly.

## 6. Datetimes

- Datetime instants: ISO-8601 / RFC 3339 with Z (UTC), millisecond
precision: 2026-05-10T18:30:00.123Z. Mirrors the substrate's
internal strftime('%Y-%m-%dT%H:%M:%fZ', 'now') format, so no
translation cost.
- Date-only fields: YYYY-MM-DD. Federal sources publish dates in
Eastern; the date *is* the Eastern calendar day. Don't promote to
midnight UTC (that would imply a precision the source doesn't have
and would shift the calendar day for late-night-ET items).
- Input: permissive (Postel's law) — accept any RFC 3339 string
including offsets, missing seconds, missing milliseconds, or
date-only. Normalize to UTC server-side.
- Convention note: when an agent needs a calendar day from a
datetime instant (e.g., late-night House votes), it applies
America/New_York. Documented; not exposed as a query param.

## 7. Filter syntax

Flat query params with explicit field names:
?status=enacted&congress=119&since=2026-01-01.

- Multi-value: comma-separated CSV — ?source=fr,bills.
- Time range: magic ?since= (inclusive) and ?until= (exclusive),
bound to the endpoint's primary time field (documented per endpoint).
- Booleans: ?is_active=true (also accepts 1).
- Reserved param names (cannot be shadowed by resource fields):
q, since, until, cursor, limit, offset, sort,
fields, include_total, source.

Operator suffixes (__gte, __in, __not), JSON:API filter brackets,
OR across fields, NOT — explicitly deferred to v1.x. Equality +
multi-value + magic time range covers ~90% of agent queries.

## 8. Sort syntax

- ?sort=-<field> for descending, ?sort=<field> for ascending.
- Multi-key: ?sort=-published_at,title.
- List endpoint default: -<primary_time_field> (newest first).
- Search endpoint default: relevance (BM25+vec hybrid, descending).
- Search + sort = top-K rerank (β semantics):
1. Hybrid scoring identifies the candidate pool of
min(1000, max(200, offset + 2*limit)) most-relevant matches.
2. Pool is re-sorted by ?sort=<field>.
3. User's offset/limit slice is returned from the re-sorted pool.

Rationale: agents searching for "defense appropriations" with
?sort=-published_at typically want "newest of the most relevant,"
not "everything matching, sorted by date." Tested by the
test_rest_conventions.py::test_search_sort_top_k_rerank case.

## 9. Fieldsets

Three implicit fieldsets per resource, documented in OpenAPI:

- card — what list and search return per row. Lightweight: id,
primary date, title, citation_string, source_url. Search adds
snippet: {text, highlights} and body: {size_bytes, url} (or null).
- full — what singleton fetch returns. card + every metadata
field + body where present. Default for GET /v1/<resource>/<id>.
- body-only — what /v1/<resource>/<id>/body returns. Just the
body fields. Convenience for clients that already have metadata cached.

?fields=<comma-list> is a whitelist; id is always included; unknown
fields produce a 400 with hint.valid_values. No ?fields=* shortcut —
singleton already returns the full record.

Snippet shape: {text, highlights: [{start, end}]}. Plain text + offsets,
not HTML-bolded. Lets clients render bolding without parsing markup.

## 10. Status codes

| Code | When |
|------|------|
| 401 | Auth failure (missing/malformed/invalid/revoked key — distinguished by error.code). |
| 403 | Authenticated but plan insufficient (code: plan_insufficient). |
| 404 | Resource never existed (record_not_found) or endpoint unknown (endpoint_not_found). |
| 410 | Resource was withdrawn / superseded upstream (record_withdrawn / record_superseded). Successor pointer in error.hint where available. |
| 429 | Rate-limit burst exceeded (rate_limit_exceeded). Retry-After header. |

Note: Josh's plan model is "subscription tier sets burst limit; no
monthly cap" — so a monthly_quota_exhausted code is NOT defined at v1.
If metered usage ships later, that adds a new code.

## 11. Rate-limit headers

On every response (success + error):
- X-RateLimit-Limit: <n>
- X-RateLimit-Remaining: <n>
- X-RateLimit-Reset: <epoch_seconds_utc>

On 429 only:
- Retry-After: <seconds>

Per-key (per-account). Window algorithm (fixed vs token bucket) is a
managed-deployment concern; conventions only commit to wire format.

## Implementation surface

- shared/josh_substrate/api/ (new subpackage when work begins) —
Pydantic models for the envelope, error, and pagination shapes.
Cursor encoding/decoding helpers. Datetime serializers.
- josh-core/josh_core/middleware/ — request-id middleware (UUIDv7
generator + X-Request-Id header on every response), error handler
that converts FastAPI/Pydantic exceptions into the spec'd envelope,
rate-limit middleware (delegates the quota check to the deployment's
auth layer; this layer only enforces shape).
- josh-core/tests/test_rest_conventions.py — the contract test
suite that the success_determiner runs.

1 of 7 done.

• t1 All 10 cross-cutting decisions resolved with rritz; spec drafted
• t2 Pydantic models for envelope + error + pagination shipped under shared/josh_substrate/api/
• t3 FastAPI middleware (request-id + error handler + rate-limit shape) shipped under josh-core/josh_core/middleware/
• t4 josh-core/tests/test_rest_conventions.py covers every acceptance criterion
• t5 OpenAPI spec emits the documented envelope/error/pagination/fieldset types
• t6 rest-api-resource-endpoints + rest-api-search + mcp-server specs declare dependency on this spec and re-use the conventions
• t7 Determiner runs green — all conventions enforced via contract tests against the live FastAPI app

• 2026-05-10T20:00:00Z (new)→planned Spec authored after a 10-question grilling-style decision pass with claude. Each cross-cutting REST decision (pagination shape, response envelope, error format, versioning style, ID format, datetime serialization, filter syntax, sort/rerank semantics, fieldset projection, status code policy, rate-limit headers) was surfaced with options + tradeoffs and locked explicitly. Decisions baked into acceptance criteria; implementation deferred to dependent specs (rest-api-resource-endpoints, rest-api-search, mcp-server) which now declare this spec as a dependency. One operational note: the rate-limit decision locked "burst-only, no monthly cap" — supersedes any plan-tier-specific rate-limit references in downstream specs.