An OCR pipeline that never acts on a number it can't prove

The problem

Checking timesheet screenshots against a leave register by hand is slow, and one misread digit sends the wrong email. Generic OCR reads a timesheet as a cloud of words and loses which number belongs to which day.

Team: Personal project. Timeline: September–October 2026.

What changed

Held-out sheets read exactly

96 of 220

Baseline: Tesseract 5, 27 of 220

220 held-out generated sheets, each with one degradation (blur, skew, noise, JPEG, shadow, low resolution, stamps, pen marks, handwriting, simulated photos), read on x86-64 in CI (docs/ocr-benchmark.md). Synthetic data only

Verified but wrong

1 of 220

A checked field wrong on a sheet the reader marked verified, same held-out set; the one miss was a handwritten correction the handwriting check didn't catch

Field accuracy

93.1%

Baseline: 80.7% before the latest fixes (development set)

Field and cell accuracy (cells 98.4%) on the same held-out synthetic set

Concurrent users

50, 0 errors

scripts/load_test.py in production mode on 2 vCPUs, 40 managers uploading and analysing while 10 users browse; 40 of 40 results correct, 0 cross-user leaks

Architecture

Decision 1 of 4

How should the reader keep each number attached to its day?

Options

Generic OCR over the whole page · A layout model · Locate the grid geometry first and read each cell on its own crop

Chose

Grid geometry first with OpenCV, then per-cell recognition

Trade-off

The reader is tuned to one timesheet layout; other scans fall back to a generic table reader and go to review more often.

Evidence

On the held-out set the reader read 96 of 220 sheets exactly, against 27 for Tesseract 5 at a median 17.9 s per sheet versus 5.8 s.

Decision 2 of 4

When is a number trusted?

Options

When OCR confidence is high · When a human approves it · When it agrees with the sheet's own row · day and grand totals

Chose

Only when it agrees with the printed totals; a uniquely determined cell is repaired, anything else goes to review

Trade-off

More sheets go to review, and pen marks that aren't data also send a sheet there.

Evidence

1 of 220 held-out sheets was verified with a wrong value; every other wrong reading was flagged rather than acted on.

Decision 3 of 4

How is a file processed exactly once?

Options

A lock in the application · A job queue's delivery guarantee · An atomic status transition in the database

Chose

UPDATE … WHERE status IN (claimable) RETURNING, with a unique run per batch

Trade-off

Batches are processed one after another per batch, so a very large batch takes about 6–10 s per sheet per OCR slot.

Evidence

Double clicks, retries and multiple replicas can't process a file twice; the 50-user load test returned 40 of 40 correct results.

Decision 4 of 4

How does OCR avoid slowing every other request?

Options

Run OCR in the request handler · Run it on the event loop · A separate low-priority thread pool

Chose

A bounded OCR thread pool at lowered CPU priority, with no database connection held while a file waits

Trade-off

Login slows under bursts, because bcrypt hashing competes for the same two CPUs.

Evidence

With OCR saturating the CPU, dashboard requests stayed at 23–30 ms p50 and 82–114 ms p95.

Claiming a file exactly once

async def claim_file(db: AsyncSession, file_id: uuid_mod.UUID) -> bool:
    result = await db.execute(
        update(UploadedFile)
        .where(UploadedFile.id == file_id, UploadedFile.status.in_(CLAIMABLE))
        .values(status=FileStatus.PROCESSING, error_message=None)
        .returning(UploadedFile.id)
    )
    await db.commit()
    return result.scalar_one_or_none() is not None


async def process_uploaded_file(db: AsyncSession, file_id: uuid_mod.UUID) -> bool:
    if not await claim_file(db, file_id):
        return False  # already being processed (or done) elsewhere

The database decides who owns a file in one statement, so retries, double clicks and extra replicas are safe without a separate lock service.

Questions?

Read the full write-up