Skip to content
← All articles
6 min read

Background jobs and queues in Go: from goroutines to reliable processing

Implementing background jobs and queues in Go often begins with a simple goroutine, but production systems require reliability guarantees that bare goroutines cannot provide. This guide covers how to choose between in-memory concurrency, relational database queues, and dedicated message brokers, alongside essential patterns for retries, idempotency, and graceful shutdown.

Why bare goroutines are insufficient for background tasks

Go makes starting asynchronous work easy: adding the go keyword spawns a goroutine with minimal overhead. However, in-memory goroutines offer no durability. If the application crashes, runs out of memory, or restarts during a deployment, all running and scheduled goroutines disappear without a trace. The caller has no visibility into whether the task completed or failed.

Uncontrolled goroutines also lack backpressure. Under a traffic surge, an HTTP handler that starts a goroutine for every request can spawn thousands of concurrent routines, exhausting database connection pools and system memory. Goroutines are appropriate for non-critical in-memory work, such as updating internal metrics, but persistent operations demand an external queue.

PostgreSQL queues with SELECT FOR UPDATE SKIP LOCKED

For many web applications, PostgreSQL is already the primary data store and can serve as an effective job queue. Using SELECT FOR UPDATE SKIP LOCKED, multiple worker instances can query the jobs table, lock available rows, and process them concurrently without blocking each other or deadlocking.

The strongest advantage of a database-backed queue is transactional consistency. You can insert a business record and its corresponding background job within a single database transaction. This eliminates the dual-write problem, guaranteeing that a job is only enqueued if the associated data changes are committed.

  • Zero additional infrastructure to operate, backup, and monitor in early stages.
  • Atomic job creation via the transactional outbox pattern.
  • Simple schema: columns for payload, status, attempt count, run_at timestamp, and locked_at.
  • Trade-off: high write volume can cause table bloat and increase WAL generation at scale.

When to introduce a dedicated message broker

A relational database queue reaches its limits when message throughput creates significant table churn, or when workloads require sub-millisecond latencies and complex routing patterns. At that stage, adopting a dedicated broker such as Redis, RabbitMQ, NATS, or Kafka becomes justified.

Brokers handle volatile throughput efficiently and provide native mechanisms for pub/sub messaging, stream persistence, and fan-out topologies. However, using a standalone broker introduces distributed systems complexity: you must handle network partitions, maintain independent cluster backups, and implement outbox patterns to prevent lost messages during dual writes.

Retries, backoff, and dead-letter handling

Network blips, third-party API rate limits, and temporary database contention cause background tasks to fail intermittently. A robust queue consumer distinguishes between transient errors, which should be retried, and permanent errors such as invalid payloads or malformed data, which will never succeed.

Retries should always incorporate exponential backoff with jitter to prevent retrying workers from synchronizing and overwhelming recovering external services. Once a task exceeds its maximum retry threshold, the consumer must route it to a dead-letter queue or mark it as dead in the database, preserving the error context for engineering review.

Designing for idempotency and at-least-once delivery

Most distributed queues guarantee at-least-once delivery, meaning a worker can receive and execute the same job more than once. If a worker completes an external action, such as dispatching a payment or sending an email, but crashes before acknowledging the job back to the queue, the task will be redelivered.

Workers must therefore be idempotent. You can achieve this by passing a unique idempotency key with each job, enforcing database unique constraints, or checking whether the intended state change has already occurred before executing side effects. If a job has already succeeded, the worker should acknowledge it and exit cleanly.

Implementing graceful shutdown in Go consumers

Deployments and container restarts issue SIGTERM or SIGINT signals to the process. Terminating a worker immediately aborts in-flight jobs, potentially leaving external resources in an inconsistent state or keeping database row locks alive until connection timeouts expire.

A Go consumer should intercept termination signals using signal.NotifyContext. Upon receiving a signal, the runner stops accepting new jobs from the queue and gives active worker goroutines a bounded period to complete their current unit of work via context.WithTimeout. If in-flight tasks cannot complete before the deadline, they should roll back their transactions and release queue locks cleanly.

Questions

FAQ

Didn't find an answer — ask on the call or in Telegram.

Can we use Go channels as an in-memory job queue?

Go channels work well for internal rate limiting and worker pool synchronization within a single process. However, they share the same durability limitations as goroutines: buffered items are lost on application restarts, and channels cannot distribute workload across multiple physical machines.

When should a team migrate from PostgreSQL queues to Redis or RabbitMQ?

Migrate when job volume begins impacting core database query latency, when PostgreSQL vacuuming cannot keep up with job table churn, or when you need specialized broker features such as delayed routing topologies and priority streams.

How should database connections be managed across Go worker goroutines?

Pass a single sql.DB instance to your worker pool rather than opening connections per job. Configure SetMaxOpenConns, SetMaxIdleConns, and SetConnMaxLifetime to align with worker concurrency and prevent connection pool starvation during processing peaks.

Related service

Go backend development for business applications

We build the server side of web products and develop existing systems. Data, traffic and reliability requirements come first, so the architecture fits the work it needs to do.

Learn about the service
Other articles
Contact

Shall we discuss your task?

Thirty minutes on a call — and you understand the architecture, timeline and engagement format. No obligations.

Or message us on Telegram →Follow our Telegram channel →

Enter your Telegram or email so we can reply.

We reply within 15 minutes.