Docs/ Spec/ substrate

substrateverifiedp1

ModalProvider — embeddings via a Modal-deployed function

embedding-provider-modal · updated — · owner rritz

Header

Use the pencil to edit title, status, priority, and owner. Changing status auto-prepends a changelog entry.

Why

GPU-backed bulk embedding via Modal's pay-per-second pricing makes the
cheapest option for large backfills (~$3 to embed 50M chunks on an
A10/L4 vs ~6 days on CPU). Adapter ships in josh-substrate and is
self-contained; the deployed Modal app spec is shipped separately so
federation users can modal deploy under their own account and pay
their own bill — Foundation doesn't middleman GPU spend.

User stories

As a deployment with a backfill to do, I want GPU-backed bulk embedding without Foundation middleman so that I can spend $3 not $300, on my own Modal account.

As an operator, I want ModalProvider to back off cleanly on cold starts and 5xx so that rate-limit blips don't burn per-job retry budget.

Acceptance criteria (EARS)

  1. When the adapter is constructed, the system shall NOT call Modal — function lookup is lazy on first `_ensure_function` call.
  2. When `embed_documents` is called, the system shall use Modal's `.remote.aio()` so the event loop is not blocked.
  3. Where Modal raises a Connection/Timeout/Unavailable-flavored exception, the adapter shall map it to `ProviderUnavailableError`; other Modal errors map to `ProviderTransientError`; lookup failures map to `ProviderConfigError`.
  4. Where the deployed Modal function returns vectors with shape ≠ `(self.dim,)`, the adapter shall raise `ProviderConfigError` (model is mis-deployed).

Success determiner

kindtest_file

Path

shared/josh_substrate/tests/embedding/test_provider_contract.py

Runner

JOSH_MODAL_TEST=1 uv run pytest shared/josh_substrate/tests/embedding -m 'contract' -k modal

Skipped by default. Set JOSH_MODAL_TEST=1 + configure ~/.modal.toml to run live. The static (mocked) verification is that the contract suite imports the adapter cleanly and the `_factory_modal` factory skips correctly when env isn't set — tested implicitly by `poe ci`.

Clarifications needed

None.

Out of scope

None.

Dependencies

Plan

shared/josh_substrate/embedding/providers/modal_provider.py. Modal
SDK in josh-substrate[modal] extra. Function lookup uses
modal.Function.from_name(app_name, function_name). Companion: a
reference Modal app spec users modal deploy under their account
(separate spec, not in scope here).

Tasks

4 of 5 done.

  • t1 ModalProvider class implementing the Protocol
  • t2 Lazy function lookup with double-checked locking
  • t3 Exception → EmbeddingError mapping (config/transient/unavailable)
  • t4 Vector-shape sanity check after each call
  • t5 Reference Modal app spec at josh-modal-embedder/ (followup spec)

Changelog

  • 2026-05-10T11:00:00Z planned→verified Adapter shipped in josh-substrate; reference Modal app deferred to follow-up.

docs/spec/embedding-provider-modal.html · generated by bin/build-spec.py