LLM
An LLM extractor that drops any task it can't quote from the message
Slack and email in, a verified and prioritised ledger of who owes what by when
Sole developer
The problem
In a small team, commitments are scattered across Slack threads and inboxes and nobody owns the list, so deadlines slip. A model that reads them can also hallucinate tasks or be manipulated by the messages it reads.
Team: Personal project. Timeline: September–October 2026.
What changed
- Item F1, held-out test
- 0.981
- Baseline: Offline rules 0.848
- GLM 5.3 Flash on 80 held-out synthetic messages, items matched one-to-one, 95% CI 0.950–1.000 by seeded bootstrap. On a harder set written by two outsiders it scored 0.734
- Item F1, outsider-written set
- 0.844
- Baseline: Offline rules 0.556
- Nemotron 3 Super, the best backend on 30 examples written to trip the tool up and committed before the changes they informed, 95% CI 0.721–0.943
- Cost per 1,000 messages (estimate)
- $0.10–$0.45
- Measured tokens multiplied by dated list prices across four hosted models; the runs themselves used free tiers
- Time to fall back in an outage
- 2.2 s
- Baseline: 44.8 s with the circuit breaker off
- The installed CLI and the real SDK against a provider simulator, with the primary provider down
Architecture
Request flows
- A message becomes a ledger item: Slack, Gmail and exports → Normalise, check sender, prefilter → Redact PII and secrets → LLM extraction (schema-constrained) → Verification (quote, owner, injection, date) → Priority scoring and lifecycle → SQLite ledger
- Provider call with fallback: Redact PII and secrets → LLM extraction (schema-constrained) → Hosted model providers
Decision 1 of 4
Who decides dates and priority, the model or code?
- Options
- Ask the model for dates and priority too · Resolve both in code
- Chose
- The model copies the deadline phrase verbatim, and code resolves the date and scores priority
- Trade-off
- The rules miss phrasings nobody wrote down.
- Evidence
- Phrases such as "24 hours before Tuesday's meeting", "by the 20th" and "kal tak" are resolved by unit-tested code, and every point of priority is shown with its reason.
Decision 2 of 4
What happens to an item the message doesn't support?
- Options
- Keep it with a low confidence · Ask a second model · Drop it unless it quotes its evidence
- Chose
- Drop it unless it quotes its evidence exactly, or fuzzily at 90+ for long quotes
- Trade-off
- Dropping an ungrounded item can lose a real task; the checks prefer precision over recall.
- Evidence
- Hallucinated quotes are dropped before the ledger, and items whose evidence addresses an AI ("note to the bot") are dropped too.
Decision 3 of 4
Read each message alone, or the whole thread?
- Options
- Send the whole thread in one call · One call per message
- Chose
- One call per message
- Trade-off
- Corrections spread across messages ("make it Tuesday") are lost.
- Evidence
- One bad message can't contaminate a whole answer and caching works per message; the outsider-written set shows the cost of the choice.
Decision 4 of 4
How does it survive a provider outage?
- Options
- Retry until the timeout · A circuit breaker with a fallback provider
- Chose
- Retries with full-jitter backoff, a circuit breaker and a second provider
- Trade-off
- Two providers to configure and keep in the evaluation.
- Evidence
- With the primary down, fallback took 2.2 s against 44.8 s without the breaker; 183 messages under injected 429 and 529 faults finished with 0 failures.
An item survives only if its quote is in the message
def is_grounded(evidence: str, text: str) -> bool:
quote, haystack = normalize(evidence).strip("\"' .…"), normalize(text)
if not quote:
return False
if quote in haystack:
return True
return (
len(quote) >= _MIN_FUZZY_LENGTH and fuzz.partial_ratio(quote, haystack) >= _FUZZY_THRESHOLD
)This check doesn't depend on the model. Whatever backend runs, an item whose evidence isn't in the message never reaches the ledger.