An LLM extractor that drops any task it can't quote from the message
Slack and email in, a verified and prioritised ledger of who owes what by when
Summary
- Problem
- In a small team, commitments are scattered across Slack threads and inboxes and nobody owns the list, so deadlines slip. A model that reads them can also hallucinate tasks or be manipulated by the messages it reads.
- My role
- Sole developer
- Team
- Personal project
- Status
- Shipped
- Timeline
- September–October 2026
- Stack
- Python, Gemini, OpenAI-compatible APIs (NVIDIA), FastAPI, SQLite, mypy, pytest
- Next
- Evaluate on a few hundred real, double-annotated messages, and strip AI-directed sentences before the model call.
What shipped
- Extracted commitments at 0.981 item F1 on 80 held-out synthetic messages (0.734 on a harder outsider-written set) by verifying every LLM output against an exact quote.
- Cut successful manipulation attempts from 5–6 to 1–2 of 10 per hosted model on the outsider-written set (defences designed after seeing it) by dropping AI-directed text in code.
- Fell back to a second provider in 2.2 s instead of 44.8 s during an outage by adding a circuit breaker and jittered retries.
- Item F1, held-out test
- 0.981
- Baseline: Offline rules 0.848
- GLM 5.3 Flash on 80 held-out synthetic messages, items matched one-to-one, 95% CI 0.950–1.000 by seeded bootstrap. On a harder set written by two outsiders it scored 0.734
- Item F1, outsider-written set
- 0.844
- Baseline: Offline rules 0.556
- Nemotron 3 Super, the best backend on 30 examples written to trip the tool up and committed before the changes they informed, 95% CI 0.721–0.943
- Cost per 1,000 messages (estimate)
- $0.10–$0.45
- Measured tokens multiplied by dated list prices across four hosted models; the runs themselves used free tiers
- Time to fall back in an outage
- 2.2 s
- Baseline: 44.8 s with the circuit breaker off
- The installed CLI and the real SDK against a provider simulator, with the primary provider down
On this page
Context and constraints
Commitments live in Slack threads and inboxes, where nobody owns the list. The tool reads those messages and keeps one checked list of who owes what by when, each item with an owner, a due date, a priority with its reasons, and the exact quote it came from, tracked across runs until a later message reports it done. It must not invent tasks, let a message set its own priority, or follow instructions hidden in the messages it reads. Personal data and secrets are masked before any API call.
Architecture
Messages are normalised, their senders checked, and obvious noise prefiltered. Personal data is redacted with stable placeholders. The model extracts items one message at a time with schema-constrained output, inside a boundary derived from the message’s own hash so the message can’t close it. Code then verifies each item: the quote must be in the message, the owner must be named in it or be its sender or a recipient, AI-directed text and claimed promises are dropped, and the deadline phrase is resolved against the send time. Completions close earlier tasks, duplicates across channels merge, and a SQLite ledger keeps state between runs. The design document covers the alternatives considered.
Request flows
- A message becomes a ledger item: Slack, Gmail and exports → Normalise, check sender, prefilter → Redact PII and secrets → LLM extraction (schema-constrained) → Verification (quote, owner, injection, date) → Priority scoring and lifecycle → SQLite ledger
- Provider call with fallback: Redact PII and secrets → LLM extraction (schema-constrained) → Hosted model providers
Engineering decisions
Who decides dates and priority, the model or code?
- Options
- Ask the model for dates and priority too
- Resolve both in code
- Chose
- The model copies the deadline phrase verbatim, and code resolves the date and scores priority
- Trade-off
- The rules miss phrasings nobody wrote down.
- Evidence
- Phrases such as "24 hours before Tuesday's meeting", "by the 20th" and "kal tak" are resolved by unit-tested code, and every point of priority is shown with its reason.
What happens to an item the message doesn't support?
- Options
- Keep it with a low confidence
- Ask a second model
- Drop it unless it quotes its evidence
- Chose
- Drop it unless it quotes its evidence exactly, or fuzzily at 90+ for long quotes
- Trade-off
- Dropping an ungrounded item can lose a real task; the checks prefer precision over recall.
- Evidence
- Hallucinated quotes are dropped before the ledger, and items whose evidence addresses an AI ("note to the bot") are dropped too.
Read each message alone, or the whole thread?
- Options
- Send the whole thread in one call
- One call per message
- Chose
- One call per message
- Trade-off
- Corrections spread across messages ("make it Tuesday") are lost.
- Evidence
- One bad message can't contaminate a whole answer and caching works per message; the outsider-written set shows the cost of the choice.
How does it survive a provider outage?
- Options
- Retry until the timeout
- A circuit breaker with a fallback provider
- Chose
- Retries with full-jitter backoff, a circuit breaker and a second provider
- Trade-off
- Two providers to configure and keep in the evaluation.
- Evidence
- With the primary down, fallback took 2.2 s against 44.8 s without the breaker; 183 messages under injected 429 and 529 faults finished with 0 failures.
Evaluation
The labelled data is synthetic and single-annotator: 70 development, 80 held-out test and 15 injection examples, plus 30 examples written by two people who hadn’t seen the code. Predictions match gold items one-to-one, with bootstrap confidence intervals. Every figure has a command that reproduces it, listed in the README.
| Backend | Held-out F1 | Outsider-written set F1 (now) | Manipulation attacks that worked, before → after |
|---|---|---|---|
| Offline rules | 0.848 | 0.600 | 2 of 10 → 0 of 10 |
| GLM 5.3 Flash | 0.981 | 0.806 | 6 of 10 → 2 of 10 |
| Gemini 3.5 Flash-Lite | 0.931 | 0.825 | 6 of 10 → 2 of 10 |
| Nemotron 3 Super | 0.863 | 0.931 | 5 of 10 → 1 of 10 |
Every backend scored lower on the outsider-written set than on the held-out split. The defences were designed after seeing those results, so the “after” figures are optimistic, and the 15-attack injection suite is a regression test rather than a held-out score because it informed the guard.
Deployment and operations
It ships as a CLI and an HTTP API, with 332 tests, 98% line and branch coverage and mypy --strict, run in CI on Python 3.11–3.14 with macOS and Windows install checks. Each message produces a JSON trace of tokens, cost, retries, fallbacks and the items each check dropped, with ids and counts but never message text. The live demo is static: it shows the real output on a synthetic sample inbox, so it costs nothing and needs no key.
Results
On 10,000 messages the offline path took 15.8 s and the hosted-model path 97 s with 0 failures. Through the HTTP API, 32 concurrent clients got a p50 of 59 ms at 500 requests a second from cache. The limits the repository states: the data is synthetic and the test split is small, so intervals are wide; the held-out split has been public, so newer models may have seen it; two manipulation attacks still get through; and the HTTP API has no authentication of its own.
Code highlight
An item survives only if its quote is in the message
This check doesn't depend on the model. Whatever backend runs, an item whose evidence isn't in the message never reaches the ledger.
def is_grounded(evidence: str, text: str) -> bool:
quote, haystack = normalize(evidence).strip("\"' .…"), normalize(text)
if not quote:
return False
if quote in haystack:
return True
return (
len(quote) >= _MIN_FUZZY_LENGTH and fuzz.partial_ratio(quote, haystack) >= _FUZZY_THRESHOLD
)What I'd do next
Label a few hundred real, double-annotated messages with inter-annotator agreement, and add a thread-aware pass so corrections spread across messages aren’t lost.
Talk about this project
I'm happy to walk through any decision here in more depth.