02/Distributed systems/2026
Relay
Durable job queue
A compact distributed job queue in Rust that makes its failure semantics visible: SQLite persistence, pull-based workers, expiring leases and fencing tokens.
Failure semantics,
made visible.
Relay is an infrastructure study, not a Redis wrapper or a dashboard. The queue provides at-least-once delivery, and says so. It does not claim exactly-once execution, because that would need a transaction spanning Relay and arbitrary user systems.
Every state change is a SQLite transaction, every lease carries a random fencing token, and the failure behavior is written down and tested.
01/Five decisions
- 01
Pull workers
Workers pull only when a semaphore slot is free, so a concurrency-8 worker holds at most eight jobs. Pulling gives natural backpressure without per-worker buffering.
- 02
Leases and fencing tokens
Time-bounded leases recover abandoned work. A random token on every lease means a worker returning after reassignment cannot acknowledge or mutate the new owner’s job.
- 03
One SQLite writer
Claims run in BEGIN IMMEDIATE transactions, so competing workers cannot both obtain a valid lease. The tradeoff is a single write coordinator and local-disk scale.
- 04
Priority aging
Strict priority can starve low-priority work. Every minute waited raises effective priority one class, so every waiting job is eventually promoted.
- 05
Bounded resources
Connections, frames, payloads and leases are capped. Jobs stay on disk, not in an in-memory channel, so overload grows queue depth, not memory.
02/Evidence
- 9,792/sLease + ACK throughput, every state change durably committed
- 270 µsp99 lease + ACK latency
- +129%Throughput after a profile-driven scheduler fix, from 4,284 jobs/s
- 7.4 MiBPeak memory with 64 worker threads
Apple M5 (10 cores, 16 GB), macOS 27.0, Rust 1.92.0. 5,000 jobs per run, 4 workers at concurrency 16. SQLite in WAL mode with synchronous=FULL. Throughput stays flat as threads increase because Relay has one transactional writer by design.
| Failure | Behavior |
|---|---|
| Worker crashes before side effect | Lease expires; another worker runs the job |
| Worker crashes after side effect, before ACK | Job can run twice; consumer idempotency is required |
| ACK is delayed past expiry | Rejected by expiry and token checks |
| Old worker returns after reassignment | Its fencing token is rejected |
| Server crashes during transaction | SQLite commits all or none of the transition |
| Server restarts with active leases | All old leases become ready; old tokens are invalid |
| Job repeatedly fails | Backoff is persisted; exhausted jobs become dead |
| Arrival exceeds processing | Durable depth rises; memory and worker leases remain bounded |
Profile, then change
The first benchmark plateaued near 4.2k lease + ACK operations per second. A profile of a 50,000-job run showed worker threads waiting on the store mutex while the claim query evaluated a computed priority expression over the whole ready set.
The fix was a partial index on ready jobs by priority and creation time. A claim now reads the oldest job in each of four priority classes, applies aging across those four candidates, and fences the winner. Scheduling semantics are unchanged. Throughput rose from 4,284 to 9,792 jobs/s and p99 latency fell from 515 to 270 µs.
What remains is dominated by serialized durable commits. Relay keeps synchronous=FULL rather than weaken durability to improve a benchmark.
Tests cover competing claims, concurrent idempotency, stale-token mutation, renewal and expiry boundaries, lost ACKs, terminal-state invariants, retries to dead letters, delayed jobs, corruption detection, bounded shutdown and a 1,000-job crash and restart recovery.
03/Limits
Not multi-region or replicated, and not a consensus system.
Not a workflow engine, a cron replacement or a sandbox for untrusted code.
The coordinator is a single process, and SQLite is its scale boundary.
Metrics counters reset on restart, and there is no persistent worker registry in v0.1.
SourceView on GitHub