Question answering over long technical documents that refuses rather than fabricates

The problem

People ask AI assistants about long handbooks, and chat models guess and fabricate sources. An answer is only useful if it points to the exact page and span it came from.

Team: Personal project. Timeline: October 2026.

What changed

Answer accuracy

0.86

19 of 22 questions on the development answer set, Gemini 3.5 Flash-Lite, judged by script (results/answer_eval.json); 22 questions is a small set

Held-out recall@5

0.83

Hybrid retrieval plus reranker on 6 held-out questions over 2 FAA documents (results/retrieval_heldout_eval.json); MRR and nDCG@5 also 0.83

Fine-tuned embeddings

No gain

Baseline: MRR 0.92 without fine-tuning

bge-small fine-tuned on 12 pairs dropped MRR to 0.83 on the same 6 held-out questions (results/finetune_retrieval.json); kept in history, not wired in

Architecture

Decision 1 of 4

How should passages be retrieved?

Options

Keyword search · Dense vectors · Both in parallel · fused by reciprocal rank and reranked

Chose

Postgres full-text and pgvector in parallel, fused with reciprocal rank fusion (k=60), then a local cross-encoder

Trade-off

Every query pays for a reranker pass.

Evidence

On the 18-question development set, keyword-only recall@5 was 0.28–0.33; the reranker raised dense recall@5 from 0.94 to 1.00 on fixed chunking.

Decision 2 of 4

Is a dedicated vector database worth it?

Options

Qdrant · Weaviate or Milvus alongside Postgres · Postgres alone with pgvector

Chose

One Postgres instance holding vectors, full-text columns, chunks, metadata and results

Trade-off

Past about a million chunks a dedicated vector index would be worth revisiting.

Evidence

The corpus is tens of documents and well under 100,000 chunks, so one store keeps deploys simple and filtering transactional.

Decision 3 of 4

How are citations trusted?

Options

Ask another model to check them · Check every quoted span mechanically

Chose

Every quoted span must appear in the retrieved text after normalisation; one repair pass, then refusal

Trade-off

A few valid answers are refused because the model paraphrased its quote.

Evidence

The failure tests (provider errors, malformed tool arguments, a dead database, a tainted PDF) all end in a refusal or degraded answer, never a fabricated one.

Decision 4 of 4

Should the embedding model be fine-tuned?

Options

Fine-tune bge-small on domain pairs · Keep the base model

Chose

Keep the base model

Trade-off

The experiment's code and result stay in history without being used.

Evidence

Fine-tuning on 12 pairs lowered held-out MRR from 0.92 to 0.83.

Checking each citation against the retrieved text

def verify_citation(citation: Citation, source_pages: dict[tuple[str, int], str]) -> bool:
    """True if the quoted span appears (whitespace/case-normalised) in the page's retrieved text."""
    source = source_pages.get((citation.document, citation.page))
    if source is None:
        return False
    return _norm(citation.quoted_span) in _norm(source)


def verify_answer(
    answer: AnswerContract, source_pages: dict[tuple[str, int], str]
) -> tuple[AnswerContract, list[str]]:
    """Return (contract, issues). Drop unverifiable citations; refuse if none
    remain on a non-refusal answer."""
    if answer.refusal:
        return answer, []
    valid: list[Citation] = []
    issues: list[str] = []
    for citation in answer.citations:
        if verify_citation(citation, source_pages):
            valid.append(citation)
        else:
            issues.append(f"span not found verbatim at {citation.document}:{citation.page}")

The check compares strings, not opinions. A citation either points at text that was actually retrieved or it is dropped, and an answer left with no valid citation becomes a refusal.

Questions?

Read the full write-up