Pavan Dangeti

A job queue where a crashed or zombie worker can't apply an effect twice

A durable job queue and workflow engine in Go, on PostgreSQL alone

SDE

Problem
Background jobs must survive crashes and restarts with no job lost and no effect applied twice. Exactly-once delivery over an unreliable network is impossible, so the guarantee has to be stated precisely.
My role
Sole developer
Team
Personal project
Status
Shipped
Timeline
October 2026
Stack
Go, PostgreSQL, TLA+ (TLC), Docker Compose, GitHub Actions
Next
Scale past the measured ceiling of about 1,200 jobs/s on one Postgres by running one Conveyor stack per queue.

What shipped

  • Applied every job's effect at most once, with 60 of 60 jobs completed and zero fencing violations under crashes, pauses, a Postgres restart and dropped connections, by guarding writes with fencing tokens.
  • Reached 1,243 jobs/s on 500 jobs, against 298–643 for River on the same machine, by batching SKIP LOCKED dequeues and debouncing LISTEN/NOTIFY wake-ups.
  • Checked the lease protocol's interleavings on every push by running a TLA+ model through TLC in CI.
Throughput, 500 jobs
1,243 jobs/s
Baseline: River 298–643 jobs/s, by fetch setting
4 worker loops × 8 in flight, debounced NOTIFY, same Mac and Postgres 17 for both (results/benchmarks-2026-10-08.json); start latency is the cost, p50 23 ms against River's 2 ms at its fastest setting
Jobs completed under faults
60 of 60
kill -9, SIGSTOP/SIGCONT, a Postgres stop and start, and 5% dropped connections at 30 ms latency (results/fault-*.json), with zero fencing-token violations
On this page
Source: docs/demo.gif in the Conveyor repository (code available on request)

Context and constraints

Background work has to survive worker crashes, network partitions and Postgres restarts, with no job lost and no effect applied twice. Conveyor guarantees two things, stated in docs/design.md §5.4: every accepted job is eventually executed at least once, and every job’s effect (a ledger insert plus a state transition) commits at most once, even when a zombie worker finishes after losing its lease. It is single-region, with no multi-tenant isolation, and it relies on Postgres’s now() for leases.

Architecture

Producers enqueue over HTTP or the Go client with an idempotency key, which is unique per queue, so a retried enqueue is harmless. Workers dequeue in batches with FOR UPDATE SKIP LOCKED, which also issues a new lease version as a fencing token. Heartbeats extend the lease only while the worker still holds the current token, and every outcome write is guarded the same way. A scheduler expires leases and runs cron, and LISTEN/NOTIFY wakes idle workers.

Architecture: ConveyorProducer (HTTP or Go client), Scheduler and lease reaper, Conveyor API, LISTEN / NOTIFY, Worker pool, Job handler and its effect, PostgreSQL (jobs, queues, cron, ledger). Connections: Producer (HTTP or Go client) to Conveyor API; Conveyor API to PostgreSQL (jobs, queues, cron, ledger); PostgreSQL (jobs, queues, cron, ledger) to LISTEN / NOTIFY; LISTEN / NOTIFY to Worker pool; Worker pool to PostgreSQL (jobs, queues, cron, ledger); Worker pool to Job handler and its effect; Job handler and its effect to PostgreSQL (jobs, queues, cron, ledger); Scheduler and lease reaper to PostgreSQL (jobs, queues, cron, ledger).enqueue with idempotency keywake-upSKIP LOCKED dequeue + fencing tokeneffect guarded by tokenexpire leases, run cronProducer (HTTP or Go client)clientScheduler and lease reaperserviceConveyor APIgatewayLISTEN / NOTIFYqueueWorker poolserviceJob handler and its effectservicePostgreSQL (jobs, queues, cron, ledger)data store

Request flows

  1. Enqueue and run a job: Producer (HTTP or Go client) → Conveyor API → PostgreSQL (jobs, queues, cron, ledger) → LISTEN / NOTIFY → Worker pool → Job handler and its effect → PostgreSQL (jobs, queues, cron, ledger)
  2. Recover from a crashed worker: Scheduler and lease reaper → PostgreSQL (jobs, queues, cron, ledger) → LISTEN / NOTIFY → Worker pool

Architecture: Conveyor

Components

Request flows

Choose a component to focus it, or a flow to watch its requests. In the view, drag or use the arrow keys to turn it and + or − to zoom. Escape closes.

Engineering decisions

Where should job state live?
Options
  • Redis
  • Kafka
  • A pluggable storage layer
  • PostgreSQL for everything
Chose
PostgreSQL for jobs, queues, cron, dependencies and the fault-injection ledger
Trade-off
The throughput ceiling is lower than Redis or Kafka; sharding by queue is the documented escape hatch, unused in v1.
Evidence
SELECT … FOR UPDATE SKIP LOCKED, advisory locks and LISTEN/NOTIFY give correctness with no second store to keep atomic with.
What does "exactly once" mean when delivery can't be exactly once?
Options
  • Claim exactly-once delivery
  • Accept duplicate effects
  • At-least-once execution with effects guarded by a fencing token
Chose
Every dequeue issues a new lease version, and every outcome write is guarded by owner and version
Trade-off
Effects in external systems are the handler's responsibility; Conveyor can only pass the token on as an idempotency key.
Evidence
A zombie worker's completion updates 0 rows, so it can't overwrite a newer result; the fault runs recorded zero fencing violations.
How are the protocol's interleavings checked?
Options
  • Unit tests only
  • Fault injection only
  • A TLA+ model checked by TLC in CI
  • plus fault injection
Chose
A TLA+ model checked in CI on a small configuration (3 jobs, 2 workers)
Trade-off
The model abstracts SQL, priorities, cron and the network, and larger configurations run by hand.
Evidence
spec/tla/Conveyor.tla runs in the CI tlc job on every push, next to go vet, staticcheck and the race-detector tests.
How do idle workers find new work quickly?
Options
  • Poll on a timer
  • NOTIFY on every enqueue
  • Debounced NOTIFY
Chose
Debounced NOTIFY
Trade-off
Start latency rises from a 1 ms p50 to 23 ms.
Evidence
Throughput rose from 789 to 1,243 jobs/s on the same 500-job run.

Evaluation

Conveyor was benchmarked against River, an established Go job queue, on the same Mac and Postgres 17. River’s throughput depends heavily on its fetch cooldown, so both settings are reported.

System Throughput p50 enqueue→start p95 p99
Conveyor, debounced NOTIFY 1,243/s 23 ms 48 ms 53 ms
Conveyor, NOTIFY per enqueue 789/s 1 ms 7 ms 13 ms
River v0.49.0, fetch cooldown 100 ms 298/s 591 ms 1,088 ms 1,165 ms
River v0.49.0, fetch cooldown 1 ms 643/s 2 ms 3 ms 5 ms

500 jobs per run. The raw outputs are in results/benchmarks-2026-10-08.json, and the fault-injection results are in results/fault-*.json (code available on request).

Deployment and operations

The full stack runs with Docker Compose. CI runs go vet, staticcheck, the tests with the race detector, and TLC on the TLA+ model on every push. Fault injection, covering kill -9, SIGSTOP, a Postgres restart and dropped connections, runs as go test ./internal/fault. Trace context is stored on every job and shown in the job view, and a runbook covers operations. The live demo is self-hosted with Docker Compose; there is no public deployment.

Results

All 60 jobs completed under every injected fault, with zero fencing-token violations. On one Postgres, throughput tops out around 1,200 jobs/s on the development Mac. River inserts faster at its default fetch settings; Conveyor trades that for an explicit fencing and heartbeat contract. The concurrency limit per queue is a soft cap, counted without locking.

Code highlight

A heartbeat that a zombie worker can't win

The lease version is the fencing token. A worker whose lease expired and was reissued matches 0 rows, learns it is stale, and must not apply its effect.

// Heartbeat extends the lease. Returns ErrStaleLease when the caller no
// longer holds the current fencing token.
func Heartbeat(ctx context.Context, db *sql.DB, id int64, owner string, version int64, lease time.Duration) error {
	res, err := db.ExecContext(ctx, `
UPDATE jobs SET lease_expiry = now() + $4::interval, updated_at=now()
WHERE id=$1 AND state='running' AND lease_owner=$2 AND lease_version=$3`,
		id, owner, version, fmt.Sprintf("%d seconds", int(lease.Seconds())))
	if err != nil {
		return err
	}
	if n, _ := res.RowsAffected(); n == 0 {
		return ErrStaleLease
	}
	return nil
}

What I'd do next

Export trace context to a tracing backend, which is stored per job today but not sent anywhere, and add more than the current minimal set of Prometheus metrics.

Talk about this project

I'm happy to walk through any decision here in more depth.

pssaikumardangeti@gmail.com