SDE
A job queue where a crashed or zombie worker can't apply an effect twice
A durable job queue and workflow engine in Go, on PostgreSQL alone
Sole developer
The problem
Background jobs must survive crashes and restarts with no job lost and no effect applied twice. Exactly-once delivery over an unreliable network is impossible, so the guarantee has to be stated precisely.
Team: Personal project. Timeline: October 2026.
What changed
- Throughput, 500 jobs
- 1,243 jobs/s
- Baseline: River 298–643 jobs/s, by fetch setting
- 4 worker loops × 8 in flight, debounced NOTIFY, same Mac and Postgres 17 for both (results/benchmarks-2026-10-08.json); start latency is the cost, p50 23 ms against River's 2 ms at its fastest setting
- Jobs completed under faults
- 60 of 60
- kill -9, SIGSTOP/SIGCONT, a Postgres stop and start, and 5% dropped connections at 30 ms latency (results/fault-*.json), with zero fencing-token violations
Architecture
Request flows
- Enqueue and run a job: Producer (HTTP or Go client) → Conveyor API → PostgreSQL (jobs, queues, cron, ledger) → LISTEN / NOTIFY → Worker pool → Job handler and its effect → PostgreSQL (jobs, queues, cron, ledger)
- Recover from a crashed worker: Scheduler and lease reaper → PostgreSQL (jobs, queues, cron, ledger) → LISTEN / NOTIFY → Worker pool
Decision 1 of 4
Where should job state live?
- Options
- Redis · Kafka · A pluggable storage layer · PostgreSQL for everything
- Chose
- PostgreSQL for jobs, queues, cron, dependencies and the fault-injection ledger
- Trade-off
- The throughput ceiling is lower than Redis or Kafka; sharding by queue is the documented escape hatch, unused in v1.
- Evidence
- SELECT … FOR UPDATE SKIP LOCKED, advisory locks and LISTEN/NOTIFY give correctness with no second store to keep atomic with.
Decision 2 of 4
What does "exactly once" mean when delivery can't be exactly once?
- Options
- Claim exactly-once delivery · Accept duplicate effects · At-least-once execution with effects guarded by a fencing token
- Chose
- Every dequeue issues a new lease version, and every outcome write is guarded by owner and version
- Trade-off
- Effects in external systems are the handler's responsibility; Conveyor can only pass the token on as an idempotency key.
- Evidence
- A zombie worker's completion updates 0 rows, so it can't overwrite a newer result; the fault runs recorded zero fencing violations.
Decision 3 of 4
How are the protocol's interleavings checked?
- Options
- Unit tests only · Fault injection only · A TLA+ model checked by TLC in CI · plus fault injection
- Chose
- A TLA+ model checked in CI on a small configuration (3 jobs, 2 workers)
- Trade-off
- The model abstracts SQL, priorities, cron and the network, and larger configurations run by hand.
- Evidence
- spec/tla/Conveyor.tla runs in the CI tlc job on every push, next to go vet, staticcheck and the race-detector tests.
Decision 4 of 4
How do idle workers find new work quickly?
- Options
- Poll on a timer · NOTIFY on every enqueue · Debounced NOTIFY
- Chose
- Debounced NOTIFY
- Trade-off
- Start latency rises from a 1 ms p50 to 23 ms.
- Evidence
- Throughput rose from 789 to 1,243 jobs/s on the same 500-job run.
A heartbeat that a zombie worker can't win
// Heartbeat extends the lease. Returns ErrStaleLease when the caller no
// longer holds the current fencing token.
func Heartbeat(ctx context.Context, db *sql.DB, id int64, owner string, version int64, lease time.Duration) error {
res, err := db.ExecContext(ctx, `
UPDATE jobs SET lease_expiry = now() + $4::interval, updated_at=now()
WHERE id=$1 AND state='running' AND lease_owner=$2 AND lease_version=$3`,
id, owner, version, fmt.Sprintf("%d seconds", int(lease.Seconds())))
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrStaleLease
}
return nil
}The lease version is the fencing token. A worker whose lease expired and was reissued matches 0 rows, learns it is stale, and must not apply its effect.