System overview
EncodeFlow separates orchestration from media execution. The Manager owns durable lifecycle decisions, Workers run FFmpeg close to compute and storage, and the Admin console turns the public Manager API into an operational view.
Components
| Component | Responsibility | Connects to |
|---|---|---|
| Manager | Validate requests, persist jobs, schedule attempts, reconcile workers, expose HTTP/SSE | PostgreSQL, NATS, OTLP collector |
| Worker | Discover capabilities, claim work, stage media, run FFmpeg, upload outputs, recover locally | NATS, S3-compatible storage |
| Admin | Monitor jobs and workers, issue cancellation and drain commands | Manager HTTP/SSE only |
| PostgreSQL | Durable jobs, attempts, workers, enrollment tokens, and outbox rows | Manager |
| JetStream | Durable work dispatch and control commands | Manager and Workers |
| Core NATS | Request/reply, heartbeats, statistics, and high-rate progress | Manager and Workers |
| S3 | Source media, outputs, manifests, and optional logs | Workers and clients |
Request-to-output flow
- 01Submit
A client sends a validated execution-plan template to the Manager.
- 02Persist
The Manager creates the job and first attempt in PostgreSQL, including a fencing token.
- 03Dispatch
An outbox publisher sends durable work to the selected JetStream pool subject.
- 04Accept
A capable Worker asks the Manager to atomically claim the current attempt.
- 05Execute
The Worker stages the input, runs FFmpeg, publishes progress, and uploads artifacts.
- 06Finalize
The Manager accepts only current fenced events and persists the terminal state.
Durable and live state
PostgreSQL is authoritative for lifecycle and ownership. A transactional outbox prevents database commits from being separated from durable NATS publication. JetStream redelivery is expected and consumers are idempotent.
Progress and Worker statistics are intentionally live data. They travel on Core NATS and are held in Manager memory, so detailed values may be null after a Manager restart until the next message arrives.
Observability path
The Manager emits OTLP/HTTP protobuf traces, metrics, and structured logs to an OpenTelemetry Collector. The local stack routes them to Tempo, Prometheus, and Loki, with Grafana as the common exploration surface.
Read the observability guide for local URLs, correlation fields, and production transport requirements.