Why bare goroutines are insufficient for background tasks
Go makes starting asynchronous work easy: adding the go keyword spawns a goroutine with minimal overhead. However, in-memory goroutines offer no durability. If the application crashes, runs out of memory, or restarts during a deployment, all running and scheduled goroutines disappear without a trace. The caller has no visibility into whether the task completed or failed.
Uncontrolled goroutines also lack backpressure. Under a traffic surge, an HTTP handler that starts a goroutine for every request can spawn thousands of concurrent routines, exhausting database connection pools and system memory. Goroutines are appropriate for non-critical in-memory work, such as updating internal metrics, but persistent operations demand an external queue.
PostgreSQL queues with SELECT FOR UPDATE SKIP LOCKED
For many web applications, PostgreSQL is already the primary data store and can serve as an effective job queue. Using SELECT FOR UPDATE SKIP LOCKED, multiple worker instances can query the jobs table, lock available rows, and process them concurrently without blocking each other or deadlocking.
The strongest advantage of a database-backed queue is transactional consistency. You can insert a business record and its corresponding background job within a single database transaction. This eliminates the dual-write problem, guaranteeing that a job is only enqueued if the associated data changes are committed.
- Zero additional infrastructure to operate, backup, and monitor in early stages.
- Atomic job creation via the transactional outbox pattern.
- Simple schema: columns for payload, status, attempt count, run_at timestamp, and locked_at.
- Trade-off: high write volume can cause table bloat and increase WAL generation at scale.
When to introduce a dedicated message broker
A relational database queue reaches its limits when message throughput creates significant table churn, or when workloads require sub-millisecond latencies and complex routing patterns. At that stage, adopting a dedicated broker such as Redis, RabbitMQ, NATS, or Kafka becomes justified.
Brokers handle volatile throughput efficiently and provide native mechanisms for pub/sub messaging, stream persistence, and fan-out topologies. However, using a standalone broker introduces distributed systems complexity: you must handle network partitions, maintain independent cluster backups, and implement outbox patterns to prevent lost messages during dual writes.
Retries, backoff, and dead-letter handling
Network blips, third-party API rate limits, and temporary database contention cause background tasks to fail intermittently. A robust queue consumer distinguishes between transient errors, which should be retried, and permanent errors such as invalid payloads or malformed data, which will never succeed.
Retries should always incorporate exponential backoff with jitter to prevent retrying workers from synchronizing and overwhelming recovering external services. Once a task exceeds its maximum retry threshold, the consumer must route it to a dead-letter queue or mark it as dead in the database, preserving the error context for engineering review.
Designing for idempotency and at-least-once delivery
Most distributed queues guarantee at-least-once delivery, meaning a worker can receive and execute the same job more than once. If a worker completes an external action, such as dispatching a payment or sending an email, but crashes before acknowledging the job back to the queue, the task will be redelivered.
Workers must therefore be idempotent. You can achieve this by passing a unique idempotency key with each job, enforcing database unique constraints, or checking whether the intended state change has already occurred before executing side effects. If a job has already succeeded, the worker should acknowledge it and exit cleanly.
Implementing graceful shutdown in Go consumers
Deployments and container restarts issue SIGTERM or SIGINT signals to the process. Terminating a worker immediately aborts in-flight jobs, potentially leaving external resources in an inconsistent state or keeping database row locks alive until connection timeouts expire.
A Go consumer should intercept termination signals using signal.NotifyContext. Upon receiving a signal, the runner stops accepting new jobs from the queue and gives active worker goroutines a bounded period to complete their current unit of work via context.WithTimeout. If in-flight tasks cannot complete before the deadline, they should roll back their transactions and release queue locks cleanly.