1,000 concurrent orders for 100 units, and exactly 100 confirmed
A distributed order system that never oversells stock, and proves it with scripts anyone can rerun
Summary
- Problem
- An order touches three databases on three services, so something eventually fails halfway. The obvious check-then-write oversold 100 units to 921–969 of 1,000 concurrent buyers.
- My role
- Sole developer
- Team
- Solo project
- Status
- Shipped
- Timeline
- September–October 2026
- Stack
- Java 21, Spring Boot 4, PostgreSQL 17, Redpanda, OpenTelemetry, Angular, Three.js, Playwright
- Next
- Run the full backend behind the live demo, which today shows the 3D ops floor in seed mode only.
What shipped
- Eliminated overselling under load, with exactly 100 of 1,000 concurrent orders confirmed for 100 units (a check-then-write baseline accepted 921–969), by using a conditional UPDATE backed by a CHECK constraint.
- Kept every cross-store invariant intact through 9 of 9 injected failure scenarios by publishing through a transactional outbox and consuming through an idempotent inbox.
- Sustained about 2,980 orders a minute at 50 orders/s offered, end-to-end p99 1.4–1.5 s on one laptop, by choreographing a four-service saga over Redpanda.
- Oversold under 1,000 concurrent orders
- 0
- Baseline: 921–969 of 1,000 told yes by a check-then-write
- 1,000 concurrent HTTP orders for 100 units at 100 clients, then nine cross-store invariants (scripts/ordering-harness.py); the baseline is measured at the reservation layer, three runs
- Failure scenarios that hold
- 9 of 9
- SIGKILL of each service mid-burst, a broker restart, duplicate and reordered deliveries, 20% declined payments and a poison message, then the invariants (scripts/chaos.sh), on 10 cores and on 2
- Throughput at 50 orders/s offered
- ~2,980/min
- k6, three runs on one Apple M5 laptop running the services, Postgres and Redpanda together; end-to-end p99 1,448–1,526 ms, and a third run hit a 15 s broker stall
On this page
Context and constraints
An order has to hold stock, take the customer’s money and claim a warehouse slot, and those live in three databases on three services. The promise is narrow and testable: stock is never oversold, no accepted order is lost, and no customer is charged twice, including when services crash mid-order. Non-goals are stated in the design document: one host, one Postgres and one broker node, simulated payments, and no distributed transactions anywhere.
Architecture
POST /api/orders commits the order and the command to reserve its stock in one transaction. An outbox relay moves that command to a Redpanda topic keyed by order number, so one order’s events arrive in sequence. Inventory holds the stock, payment takes the money, fulfilment claims a slot and order-service confirms. A refused step fails the order and broadcasts one compensation request, and each service undoes its own effect. Each service has its own Postgres schema and login role, so no service can read another’s tables.
Request flows
- Place an order: Seller app and CS console → order-service → Outbox and inbox tables → Redpanda topic (keyed by order) → inventory-service → Postgres (schema per service)
- Take payment: Redpanda topic (keyed by order) → payment-service → Postgres (schema per service)
- Claim a warehouse slot: Redpanda topic (keyed by order) → fulfilment-service → Postgres (schema per service)
Engineering decisions
How is stock never promised twice?
- Options
- Read
- check and write in application code
- Pessimistic row locks in application code
- One conditional UPDATE with a CHECK constraint behind it
- Chose
- One conditional UPDATE with a CHECK constraint behind it
- Trade-off
- The rule lives in SQL rather than in the domain code, so it is less visible to someone reading only the Java.
- Evidence
- The check-then-write told 300 of 300 contenders yes for 100 units; the conditional UPDATE gave exactly 100 reservations and 0 oversold.
How do events stay consistent with the data that caused them?
- Options
- Publish after committing
- Publish from the outbox table alone
- Write an outbox row in the same transaction and relay it to Redpanda
- Chose
- An outbox row in the same transaction, relayed to Redpanda
- Trade-off
- Delivery becomes at-least-once, so every consumer has to be idempotent, and the single relay thread is the throughput ceiling.
- Evidence
- All nine failure scenarios, including killing each service mid-order and restarting the broker, end with every invariant holding.
How are duplicate deliveries made harmless?
- Options
- Deduplicate in Redis
- Rely on unique constraints on each effect
- An inbox table claimed in the consumer's own transaction
- Chose
- An inbox table claimed in the same transaction as the effect
- Trade-off
- Every service keeps an inbox table that grows with the events it has handled.
- Evidence
- The chaos run re-delivers 30 duplicates and 10 reordered messages; no customer is charged twice.
Who undoes a step when the order fails?
- Options
- An orchestrator service
- A central compensating command per effect
- Each service undoes its own effect
- Chose
- Each service undoes its own effect, keyed by order number
- Trade-off
- With no orchestrator, the order's progress is spread across four services' tables.
- Evidence
- A release that arrives before its reserve leaves a tombstone, so the late reserve is refused instead of holding stock for a dead order.
Evaluation
Every number comes from a script in the repository, listed with the command that produced it in the README results table. The only honest baseline for a bespoke order saga is the obvious implementation, so the oversell test measures a read-check-write version against the shipped one.
| Result | How it was measured |
|---|---|
| 300 orders for 100 units: 100 reserved, 0 oversold (check-then-write: 300 of 300 told yes) | 300 contenders on a 16-thread pool |
| 1,000 orders for 100 units: 100 confirmed, 900 rejected (check-then-write: 921–969 told yes) | 1,000 concurrent HTTP orders, then nine invariants |
| All 9 failure scenarios hold, with 263–300 orders caught mid-saga in each crash | docs/ordering-invariants.md |
| ~2,980 orders/min at 50/s offered, end-to-end p99 1,448–1,526 ms | k6, three runs, docs/performance.md |
| One order traced as 13 spans across 4 services, confirmed in 445–801 ms | scripts/trace-order.sh, four runs |
“End-to-end” runs from the client’s POST to its first observation of a terminal saga step, polled every 200 ms, so that polling interval is inside every figure.
Deployment and operations
The whole stack runs from one compose.yaml, and a GitHub Codespace runs all of it in a browser tab. CI runs 99 Java tests against real Postgres with Testcontainers, Pact consumer-driven contracts, 15 Playwright tests and the ordering invariants on every push, with failure injection on demand. An observability profile adds OpenTelemetry, Jaeger, Prometheus and Grafana. The live demo is the 3D ops floor on seed data only: five JVMs, Postgres and Redpanda need more memory than any free tier offers.
Results
The oversell guarantee holds under 1,000 concurrent orders and under every failure scenario tested, and it was reproduced on a second machine. The throughput figures are a property of one laptop: in one of six load-test runs every service’s sends to the single broker timed out together for about 15 seconds. Nothing was lost, but nothing moved. Tracing was sampled at 100% during the load test, so its cost is inside those numbers.
Limitations the repository states: payment declines are simulated, alerts are raised but nothing pages anyone, a shipped order still counts as holding its reservation, the relay is one thread, reconciliation re-reads every unconfirmed order on every pass, and the signing key is generated at startup, so order-service can’t run as several replicas.
Code highlight
The reservation that cannot oversell
One statement decides the outcome. The row lock makes a competing reserve wait and re-check against the committed value, and the price comes from the row that was updated, so the amount charged can't be set by the caller.
@Transactional
public Optional<BigDecimal> reserve(String orderNumber, String sku, int quantity) {
lock(orderNumber);
String held = statusOf(orderNumber);
if ("RELEASED".equals(held)) {
return Optional.empty();
}
if ("RESERVED".equals(held)) {
return unitPriceOf(sku);
}
// One statement decides the outcome: the row lock makes a concurrent reserve wait, and on
// waking it re-checks the condition against the committed value, so units are never
// promised twice. The CHECK constraint on products is the backstop if this is ever wrong.
Optional<BigDecimal> price = jdbc.sql("""
update products set reserved = reserved + :qty
where sku = :sku and reserved + :qty <= on_hand
returning unit_price""")
.param("qty", quantity)
.param("sku", sku)
.query(BigDecimal.class)
.optional();
if (price.isEmpty()) {
return Optional.empty();
}What I'd do next
Make warehouse capacity configurable and alert on the fulfilment rejection rate: the load test failed 219 of 1,200 orders on slots while 500,000 units were in stock, which is correct behaviour but a silent ceiling.
Talk about this project
I'm happy to walk through any decision here in more depth.