Workers consume from a queue and run jobs. Use a dedicated small instance or run alongside the app. Scale workers horizontally; ensure idempotency and dead-letter handling. Monitor queue depth and processing time.
What workers do
- Consume: Pull messages from a queue (Redis list/stream, RabbitMQ, SQS, etc.). Each message triggers a job (e.g. send email, process image, sync data).
- Decouple: App enqueues work and returns; workers process async. Handles spikes and long-running tasks without blocking the web tier.
- Backends: Redis (simple queues or streams), RabbitMQ (routing, DLQ), SQS (managed, at-scale). Choose by latency, durability, and ops preference.
Deployment options
- Dedicated instance(s): Small VPS or container(s) that only run workers. Isolates CPU/memory from the app; easier to scale workers independently. Good when job volume is high or jobs are heavy.
- Alongside app: Workers on the same box as the app. Simpler for small setups; one less server. Risk: heavy jobs can starve the app (CPU, I/O). Use process manager (systemd, supervisord) and resource limits.
- Serverless: Lambda, Cloud Run, or similar for event-driven jobs. Pay per invocation; no idle cost. Suits sporadic or event-triggered workloads; cold start and timeout limits apply.
Reliability
- Idempotency: Design jobs so running twice (e.g. retry after timeout) does not cause duplicate side effects. Use idempotency keys or natural keys (e.g. "update user X") and check before applying.
- Dead-letter: When a job fails after N retries, move it to a dead-letter queue (DLQ). Monitor DLQ; fix bad messages or replay after fixing the bug. Prevents one bad message from blocking the queue.
- Visibility: Monitor queue depth (backlog), processing time, and error rate. Alert on growing backlog or spike in failures. Scale workers up/down or fix consumers.
Summary
Workers consume from a queue and run jobs. Use a dedicated instance or run alongside the app. Scale horizontally; ensure idempotency and dead-letter handling. Monitor queue depth and processing time.
