Connect API and worker
Queue, storage, and deployment boundaries for your own service.
This page describes your own deployment. For managed access, use the hosted API quickstart.
The worker repository includes a queue consumer, not an HTTP server. Supply a compatible job producer and storage integration when running your own service. A partner API key gives HTTP access to the hosted service, not access to its queue or infrastructure.
Components and boundaries
| Component | Responsibility |
|---|---|
| API + PostgreSQL | Authentication, ownership, persistent job and file records |
| Redis + BullMQ | Dispatch, progress, cancellation, and worker heartbeat |
| Worker | Extraction, validation, processing, and storage |
| Storage | Private objects, plus a separate public bucket when used |
API and worker must agree on Redis URL, queue prefix, storage configuration, and job contract version. API replicas sharing a database must also share the queue. Isolate staging with a separate database and queue namespace; a different queue prefix alone is insufficient.
Queue consumer
After building the worker, configure its private environment and run pnpm start. This launches dist/queue-worker.js. The public repository provides .env.example and deployment examples for the supported backends.
Partners call HTTP endpoints. Do not expose Redis to them, or put connection strings and storage credentials in browser code or Git.
Storage and retention
LocalStorageAdapter writes to a controlled directory and supports private delivery. Public delivery requires a dedicated R2 bucket/CDN. In a split local-storage deployment, the worker writes a shared volume and the API reads it through authenticated file routes.
The R2 adapter uses create-only identities to recover stored results after redelivery. Custom storage adapters must implement compatible lookup and identity behavior if you rely on that recovery.
Configure retention explicitly. Queue retention and temporary-directory cleanup do not delete every stored artifact.
Source credentials and providers
Source authentication belongs to the executing worker, not a public HTTP body. Keep authorized site cookies outside the checkout, mount the host file read-only, and never forward them to public third-party resolvers. Optional providers need a permission and privacy review before enabling them.
Changing execution settings on the API process does not configure a separate worker.
Readiness and isolation
GET /api/v1/health/ready checks PostgreSQL, Redis, and worker heartbeat. It does not verify every external source.
Use non-root execution, bounded CPU/memory/disk, appropriate network restrictions, and tested backups. Review the worker's reliability notes and security policy.