SDE
1,000 concurrent orders for 100 units, and exactly 100 confirmed
A distributed order system that never oversells stock, and proves it with scripts anyone can rerun
Sole developer
The problem
An order touches three databases on three services, so something eventually fails halfway. The obvious check-then-write oversold 100 units to 921–969 of 1,000 concurrent buyers.
Team: Solo project. Timeline: September–October 2026.
What changed
- Oversold under 1,000 concurrent orders
- 0
- Baseline: 921–969 of 1,000 told yes by a check-then-write
- 1,000 concurrent HTTP orders for 100 units at 100 clients, then nine cross-store invariants (scripts/ordering-harness.py); the baseline is measured at the reservation layer, three runs
- Failure scenarios that hold
- 9 of 9
- SIGKILL of each service mid-burst, a broker restart, duplicate and reordered deliveries, 20% declined payments and a poison message, then the invariants (scripts/chaos.sh), on 10 cores and on 2
- Throughput at 50 orders/s offered
- ~2,980/min
- k6, three runs on one Apple M5 laptop running the services, Postgres and Redpanda together; end-to-end p99 1,448–1,526 ms, and a third run hit a 15 s broker stall
Architecture
Request flows
- Place an order: Seller app and CS console → order-service → Outbox and inbox tables → Redpanda topic (keyed by order) → inventory-service → Postgres (schema per service)
- Take payment: Redpanda topic (keyed by order) → payment-service → Postgres (schema per service)
- Claim a warehouse slot: Redpanda topic (keyed by order) → fulfilment-service → Postgres (schema per service)
Decision 1 of 4
How is stock never promised twice?
- Options
- Read · check and write in application code · Pessimistic row locks in application code · One conditional UPDATE with a CHECK constraint behind it
- Chose
- One conditional UPDATE with a CHECK constraint behind it
- Trade-off
- The rule lives in SQL rather than in the domain code, so it is less visible to someone reading only the Java.
- Evidence
- The check-then-write told 300 of 300 contenders yes for 100 units; the conditional UPDATE gave exactly 100 reservations and 0 oversold.
Decision 2 of 4
How do events stay consistent with the data that caused them?
- Options
- Publish after committing · Publish from the outbox table alone · Write an outbox row in the same transaction and relay it to Redpanda
- Chose
- An outbox row in the same transaction, relayed to Redpanda
- Trade-off
- Delivery becomes at-least-once, so every consumer has to be idempotent, and the single relay thread is the throughput ceiling.
- Evidence
- All nine failure scenarios, including killing each service mid-order and restarting the broker, end with every invariant holding.
Decision 3 of 4
How are duplicate deliveries made harmless?
- Options
- Deduplicate in Redis · Rely on unique constraints on each effect · An inbox table claimed in the consumer's own transaction
- Chose
- An inbox table claimed in the same transaction as the effect
- Trade-off
- Every service keeps an inbox table that grows with the events it has handled.
- Evidence
- The chaos run re-delivers 30 duplicates and 10 reordered messages; no customer is charged twice.
Decision 4 of 4
Who undoes a step when the order fails?
- Options
- An orchestrator service · A central compensating command per effect · Each service undoes its own effect
- Chose
- Each service undoes its own effect, keyed by order number
- Trade-off
- With no orchestrator, the order's progress is spread across four services' tables.
- Evidence
- A release that arrives before its reserve leaves a tombstone, so the late reserve is refused instead of holding stock for a dead order.
The reservation that cannot oversell
@Transactional
public Optional<BigDecimal> reserve(String orderNumber, String sku, int quantity) {
lock(orderNumber);
String held = statusOf(orderNumber);
if ("RELEASED".equals(held)) {
return Optional.empty();
}
if ("RESERVED".equals(held)) {
return unitPriceOf(sku);
}
// One statement decides the outcome: the row lock makes a concurrent reserve wait, and on
// waking it re-checks the condition against the committed value, so units are never
// promised twice. The CHECK constraint on products is the backstop if this is ever wrong.
Optional<BigDecimal> price = jdbc.sql("""
update products set reserved = reserved + :qty
where sku = :sku and reserved + :qty <= on_hand
returning unit_price""")
.param("qty", quantity)
.param("sku", sku)
.query(BigDecimal.class)
.optional();
if (price.isEmpty()) {
return Optional.empty();
}One statement decides the outcome. The row lock makes a competing reserve wait and re-check against the committed value, and the price comes from the row that was updated, so the amount charged can't be set by the caller.