An OCR pipeline that never acts on a number it can't prove
Timesheet screenshots in, verified leave compliance out, on synthetic data
Summary
- Problem
- Checking timesheet screenshots against a leave register by hand is slow, and one misread digit sends the wrong email. Generic OCR reads a timesheet as a cloud of words and loses which number belongs to which day.
- My role
- Sole developer
- Team
- Personal project
- Status
- Shipped
- Timeline
- September–October 2026
- Stack
- Python, FastAPI, PostgreSQL, RapidOCR (PP-OCRv4), OpenCV, React, TypeScript, Tailwind CSS
- Next
- Benchmark on real captures, including phone photos, which have not been tested.
What shipped
- Read 96 of 220 held-out synthetic timesheets exactly, against 27 for Tesseract 5, by locating the grid with OpenCV and recognising each cell on its own crop.
- Limited silent errors to 1 of 220 held-out synthetic sheets verified with a wrong value by reconciling every reading against the sheet's own totals.
- Served 50 concurrent users with 0 errors and 40 of 40 correct results on 2 vCPUs by claiming work atomically and running OCR in a low-priority thread pool.
- Held-out sheets read exactly
- 96 of 220
- Baseline: Tesseract 5, 27 of 220
- 220 held-out generated sheets, each with one degradation (blur, skew, noise, JPEG, shadow, low resolution, stamps, pen marks, handwriting, simulated photos), read on x86-64 in CI (docs/ocr-benchmark.md). Synthetic data only
- Verified but wrong
- 1 of 220
- A checked field wrong on a sheet the reader marked verified, same held-out set; the one miss was a handwritten correction the handwriting check didn't catch
- Field accuracy
- 93.1%
- Baseline: 80.7% before the latest fixes (development set)
- Field and cell accuracy (cells 98.4%) on the same held-out synthetic set
- Concurrent users
- 50, 0 errors
- scripts/load_test.py in production mode on 2 vCPUs, 40 managers uploading and analysing while 10 users browse; 40 of 40 results correct, 0 cross-user leaks
- Demo
- Code available on request
- Present this project
On this page
Context and constraints
Timesheet pages are rendered by a web app, so their layout is regular even when the data isn’t. The rule the system is built around is that a number that can’t be proven is never acted on. All data in the repository, the screenshots and the benchmark is synthetic, generated by a script with invented people and projects; no real employee data is used or committed.
Architecture
A screenshot is levelled and its scale normalised from the size of the day boxes. OpenCV finds the table rulings and day boxes, which gives the exact cell grid at any browser zoom. Text columns are read with PP-OCRv4 through RapidOCR, each inked day cell is recognised on its own crop, and its fill colour gives its status. Every value is then reconciled against the row, day and grand totals: a sheet where everything agrees is verified, a uniquely determined cell is repaired and reported, and anything ambiguous goes to review. The FastAPI backend runs OCR in its own low-priority thread pool and claims work with atomic status transitions in PostgreSQL.
Request flows
- Upload and verify a sheet: React app → Vercel static build and /api rewrite → FastAPI (JWT, RBAC, rate limits) → Job dispatch → OCR reader and reconciler → File storage (disk or S3)
- Claim a file exactly once: FastAPI (JWT, RBAC, rate limits) → Job dispatch → PostgreSQL
Engineering decisions
How should the reader keep each number attached to its day?
- Options
- Generic OCR over the whole page
- A layout model
- Locate the grid geometry first and read each cell on its own crop
- Chose
- Grid geometry first with OpenCV, then per-cell recognition
- Trade-off
- The reader is tuned to one timesheet layout; other scans fall back to a generic table reader and go to review more often.
- Evidence
- On the held-out set the reader read 96 of 220 sheets exactly, against 27 for Tesseract 5 at a median 17.9 s per sheet versus 5.8 s.
When is a number trusted?
- Options
- When OCR confidence is high
- When a human approves it
- When it agrees with the sheet's own row
- day and grand totals
- Chose
- Only when it agrees with the printed totals; a uniquely determined cell is repaired, anything else goes to review
- Trade-off
- More sheets go to review, and pen marks that aren't data also send a sheet there.
- Evidence
- 1 of 220 held-out sheets was verified with a wrong value; every other wrong reading was flagged rather than acted on.
How is a file processed exactly once?
- Options
- A lock in the application
- A job queue's delivery guarantee
- An atomic status transition in the database
- Chose
- UPDATE … WHERE status IN (claimable) RETURNING, with a unique run per batch
- Trade-off
- Batches are processed one after another per batch, so a very large batch takes about 6–10 s per sheet per OCR slot.
- Evidence
- Double clicks, retries and multiple replicas can't process a file twice; the 50-user load test returned 40 of 40 correct results.
How does OCR avoid slowing every other request?
- Options
- Run OCR in the request handler
- Run it on the event loop
- A separate low-priority thread pool
- Chose
- A bounded OCR thread pool at lowered CPU priority, with no database connection held while a file waits
- Trade-off
- Login slows under bursts, because bcrypt hashing competes for the same two CPUs.
- Evidence
- With OCR saturating the CPU, dashboard requests stayed at 23–30 ms p50 and 82–114 ms p95.
Evaluation
The benchmark uses 220 held-out generated sheets, each with one degradation, read on x86-64, alongside the 220-sheet development set the fixes were tuned on.
| Metric | Held-out | Development | Before the latest fixes (development) |
|---|---|---|---|
| Sheets read exactly | 96 / 220 (43.6%) | 86 / 220 (39.1%) | 16 / 220 (7.3%) |
| Verified but wrong | 1 / 220 | 0 / 220 | 1 / 220 |
| Field accuracy · cell accuracy | 93.1% · 98.4% | 90.5% · 98.0% | 80.7% · 96.0% |
Tesseract 5 read 27 of the 220 held-out sheets exactly, with none verified but wrong. The backend suite has 336 tests against real PostgreSQL, including a permission matrix of 42 endpoints by four roles with 0 leaks, and 48 end-to-end tests in a real browser. Source: docs/ocr-benchmark.md and the README’s test results, in the repository (available on request).
Deployment and operations
The front end is a static build on Vercel, which proxies /api to the FastAPI service so the refresh cookie stays first-party. Access tokens are short-lived, refresh tokens rotate with reuse detection, and interrupted batches resume on startup. CI runs lint, the backend suite against PostgreSQL, and the front end’s type check, tests and build on every push; the OCR benchmark runs as its own workflow.
Results
On synthetic sheets the reader verifies most of what it reads correctly and refuses what it can’t prove: one held-out sheet in 220 was verified with a wrong value. The limits are stated in the repository. Real phone photos have not been tested, and simulated photos are all refused. OCR output differs between processor architectures, so published numbers come from x86-64 runs. Project and task codes, names and titles are not cross-checked against anything.
Code highlight
Claiming a file exactly once
The database decides who owns a file in one statement, so retries, double clicks and extra replicas are safe without a separate lock service.
async def claim_file(db: AsyncSession, file_id: uuid_mod.UUID) -> bool:
result = await db.execute(
update(UploadedFile)
.where(UploadedFile.id == file_id, UploadedFile.status.in_(CLAIMABLE))
.values(status=FileStatus.PROCESSING, error_message=None)
.returning(UploadedFile.id)
)
await db.commit()
return result.scalar_one_or_none() is not None
async def process_uploaded_file(db: AsyncSession, file_id: uuid_mod.UUID) -> bool:
if not await claim_file(db, file_id):
return False # already being processed (or done) elsewhereWhat I'd do next
Measure the reader on real captures, starting with phone photos, and only then decide whether photo support is worth claiming.
Talk about this project
I'm happy to walk through any decision here in more depth.