Question answering over long technical documents that refuses rather than fabricates
Answers about long technical handbooks with citations checked against the exact retrieved text
Summary
- Problem
- People ask AI assistants about long handbooks, and chat models guess and fabricate sources. An answer is only useful if it points to the exact page and span it came from.
- My role
- Sole developer
- Team
- Personal project
- Status
- Shipped
- Timeline
- October 2026
- Stack
- Python, FastAPI, PostgreSQL (pgvector), Cross-encoder reranker, Gemini, Docker
- Next
- Publish the multi-model comparison, which the slow free-tier endpoint currently blocks.
What shipped
- Answered 19 of 22 development questions correctly (0.86, a small set) by combining hybrid retrieval, reranking and a refusal when a quoted span isn't in the retrieved text.
- Reached recall@5 of 0.83 on 6 held-out questions by fusing Postgres full-text and pgvector results with reciprocal rank fusion and a cross-encoder reranker.
- Refused rather than fabricated in every failure mode its 44 tests cover by verifying each citation mechanically before answering.
- Answer accuracy
- 0.86
- 19 of 22 questions on the development answer set, Gemini 3.5 Flash-Lite, judged by script (results/answer_eval.json); 22 questions is a small set
- Held-out recall@5
- 0.83
- Hybrid retrieval plus reranker on 6 held-out questions over 2 FAA documents (results/retrieval_heldout_eval.json); MRR and nDCG@5 also 0.83
- Fine-tuned embeddings
- No gain
- Baseline: MRR 0.92 without fine-tuning
- bge-small fine-tuned on 12 pairs dropped MRR to 0.83 on the same 6 held-out questions (results/finetune_retrieval.json); kept in history, not wired in
- Code available on request
- Present this project
On this page
Context and constraints
SpecPilot answers questions about long technical documents, using FAA and NASA-style public-domain handbooks as the corpus. It runs a plan-and-act agent over a small tool set (search, read a section, look up a table, calculate, compare documents), answers only when it can point to the exact page and quoted span, and treats document text as data, never as instructions. Answers are limited to what that corpus supports, by design.
Architecture
PDFs are parsed with tables, captions and an OCR fallback, chunked, and indexed in one Postgres instance with pgvector for dense search and full-text columns for keyword search. A question runs both searches in parallel, fuses them by reciprocal rank and reranks with a local cross-encoder. The agent plans tool calls, the model drafts an answer contract with citations, and a deterministic check verifies every quoted span against the retrieved text before anything is returned, streamed over server-sent events.
Request flows
- Retrieve and rerank: Web UI → FastAPI (/search, /ask over SSE) → Plan-and-act agent and tools → Postgres (pgvector + full-text)
- Answer only with verified citations: FastAPI (/search, /ask over SSE) → Plan-and-act agent and tools → Hosted LLM
- Verify before answering: Plan-and-act agent and tools → Deterministic citation check → FastAPI (/search, /ask over SSE)
Engineering decisions
How should passages be retrieved?
- Options
- Keyword search
- Dense vectors
- Both in parallel
- fused by reciprocal rank and reranked
- Chose
- Postgres full-text and pgvector in parallel, fused with reciprocal rank fusion (k=60), then a local cross-encoder
- Trade-off
- Every query pays for a reranker pass.
- Evidence
- On the 18-question development set, keyword-only recall@5 was 0.28–0.33; the reranker raised dense recall@5 from 0.94 to 1.00 on fixed chunking.
Is a dedicated vector database worth it?
- Options
- Qdrant
- Weaviate or Milvus alongside Postgres
- Postgres alone with pgvector
- Chose
- One Postgres instance holding vectors, full-text columns, chunks, metadata and results
- Trade-off
- Past about a million chunks a dedicated vector index would be worth revisiting.
- Evidence
- The corpus is tens of documents and well under 100,000 chunks, so one store keeps deploys simple and filtering transactional.
How are citations trusted?
- Options
- Ask another model to check them
- Check every quoted span mechanically
- Chose
- Every quoted span must appear in the retrieved text after normalisation; one repair pass, then refusal
- Trade-off
- A few valid answers are refused because the model paraphrased its quote.
- Evidence
- The failure tests (provider errors, malformed tool arguments, a dead database, a tainted PDF) all end in a refusal or degraded answer, never a fabricated one.
Should the embedding model be fine-tuned?
- Options
- Fine-tune bge-small on domain pairs
- Keep the base model
- Chose
- Keep the base model
- Trade-off
- The experiment's code and result stay in history without being used.
- Evidence
- Fine-tuning on 12 pairs lowered held-out MRR from 0.92 to 0.83.
Evaluation
The evaluation sets are small, so these numbers say more about the method than about general performance.
| Measure | Result | Set |
|---|---|---|
| Recall@5, MRR, nDCG@5 (hybrid + reranker, fixed chunking) | 1.00, 1.00, 1.00 | 18 development questions, used while building |
| Same, section or recursive chunking | 0.94, 0.92, 0.92 | 18 development questions |
| Recall@5, MRR, nDCG@5, held out | 0.83, 0.83, 0.83 | 6 questions on 2 FAA documents |
| Answer accuracy | 0.86 (19 of 22) | 22 development questions |
| Search latency p50 / p95 | 0.288 s / 0.431 s | 25 sequential calls on an Apple M5 |
All figures come from committed scripts, with raw JSON under results/ (code available on request).
Deployment and operations
The service ships as a Docker image with a web UI and a FastAPI backend. CI runs 44 tests, including failure modes: provider errors and timeouts, malformed tool arguments, a database that dies during a tool call, and a tainted synthetic PDF, end to end. Dependencies are audited with pip-audit, and a runbook covers operations.
Results
The pipeline refuses rather than fabricates in every failure mode tested, and its citations are checked mechanically. Held-out retrieval reached 0.83 on six questions, below the development-set figures, which were measured on the questions used while building. Fine-tuning the embedding model made retrieval worse and was left out. A comparison across more models is blocked because the free-tier endpoint for the second provider takes 60–150 s per call.
Code highlight
Checking each citation against the retrieved text
The check compares strings, not opinions. A citation either points at text that was actually retrieved or it is dropped, and an answer left with no valid citation becomes a refusal.
def verify_citation(citation: Citation, source_pages: dict[tuple[str, int], str]) -> bool:
"""True if the quoted span appears (whitespace/case-normalised) in the page's retrieved text."""
source = source_pages.get((citation.document, citation.page))
if source is None:
return False
return _norm(citation.quoted_span) in _norm(source)
def verify_answer(
answer: AnswerContract, source_pages: dict[tuple[str, int], str]
) -> tuple[AnswerContract, list[str]]:
"""Return (contract, issues). Drop unverifiable citations; refuse if none
remain on a non-refusal answer."""
if answer.refusal:
return answer, []
valid: list[Citation] = []
issues: list[str] = []
for citation in answer.citations:
if verify_citation(citation, source_pages):
valid.append(citation)
else:
issues.append(f"span not found verbatim at {citation.document}:{citation.page}")What I'd do next
Grow the held-out question set well beyond six questions, so the retrieval and answer numbers can be compared with confidence.
Talk about this project
I'm happy to walk through any decision here in more depth.