Measure first, then improve the pipeline
The experiments were intentionally staged: establish a dense baseline, inspect its failures, reduce policy-space noise, test reranking, and only then evaluate answer generation.
Fully local baseline
Embeddings: nomic-embed-text (768-D)
Vector DB: ChromaDB
Generator: qwen3:0.6b
Local judge: qwen2.5:1.5b-instruct + RAGAS
Retrieval before generation
Ground truth stores exact source evidence, allowing Hit@K, Recall@K, Precision@K and MRR to be measured without depending on the generator.
CPU-conscious
Latency was measured because an improvement that is too expensive on CPU is not automatically the preferred system.
Dense baseline, then chunking and failure analysis
The first 20-question benchmark showed that Top-3 coverage was already strong, but additional chunks often came from the wrong policy.
Initial dense retrieval · 20 questions
| Metric | K=1 | K=3 | K=5 |
|---|---|---|---|
| Hit@K | 0.800 | 1.000 | 1.000 |
| Recall@K | 0.775 | 1.000 | 1.000 |
| Precision@K | 0.800 | 0.367 | 0.220 |
| MRR | — | — | 0.883 |
Interpretation: the required evidence was normally present by Top-3, but extra retrieved chunks diluted precision.
Chunking experiment
| Size / overlap | Chunks | Hit@1 | Recall@1 | Recall@3 | Prec@3 | MRR@5 |
|---|---|---|---|---|---|---|
| 400 / 80 | 38 | 0.800 | 0.725 | 0.950 | 0.383 | 0.892 |
| 700 / 120 | 22 | 0.800 | 0.775 | 1.000 | 0.367 | 0.883 |
| 1000 / 150 | 16 | 0.700 | 0.650 | 1.000 | 0.367 | 0.825 |
700 / 120 was selected: complete Recall@3 with fewer chunks than 400/80; 1000/150 reduced first-rank quality.
Policy routing fixed the main source-level error
TF-IDF was used only to select likely policy files. Dense Nomic retrieval still ranked chunks inside that candidate search space.
20-question source-cleanliness result
≈87% reduction in wrong-policy chunks. Questions containing any wrong-policy chunk dropped from 11/20 to 1/20.
Why a new metric was needed
Exact Evidence Precision@3 stayed at 0.367, because a correct-policy chunk can be useful without containing the exact evidence string. Source Precision@3 exposed the improvement:
Ranking on the original 20 questions
Because all 20 questions already had correct evidence inside Top-3, the benchmark was too easy to meaningfully study reranking. This motivated a harder 100-question set.
Harder 100-question retrieval benchmark
The benchmark deliberately includes paraphrases, scenarios, thresholds, negative conditions, location rules, approval conditions, historical/current distinctions and cross-policy questions.
Final three-system comparison · Top-3
| Metric | Dense only | TF-IDF + dense | TF-IDF + dense + reranker |
|---|---|---|---|
| Hit@1 | 0.730 | 0.790 | 0.840 |
| Hit@3 | 0.990 | 1.000 | 1.000 |
| Recall@3 | 0.938 | 0.938 | 0.947 |
| Evidence Precision@3 | 0.420 | 0.413 | 0.420 |
| Source Precision@3 | 0.847 | 0.950 | 0.963 |
| MRR@3 | 0.850 | 0.888 | 0.917 |
Ranking-quality snapshot
The cross-encoder improved first-relevant ranking for 11 questions, worsened 7, and left 82 unchanged.
CPU latency trade-off
| Stage | Average |
|---|---|
| Nomic query embedding | 64.32 ms |
| Dense after embedding | 1.76 ms |
| TF-IDF + dense after embedding | 2.97 ms |
| TF-IDF + dense + reranker after embedding (cross-encoder/ms-marco-MiniLM-L6-v2) | 401.08 ms |
Approximate total retrieval: ~66 ms dense, ~67 ms routed dense, ~465 ms reranked.
Grounded generation and judge comparison
The generator was deliberately kept small so that retrieval and grounding quality remained visible rather than being masked by a large model's prior knowledge.
Final four-case generation comparison · 100 questions
| Pipeline | Judge | Faithfulness | Factual correctness | Semantic similarity | Answer relevancy |
|---|---|---|---|---|---|
| Dense | RAGAS + Qwen2.5 | 0.801 | 0.652 | 0.779 | — |
| TF-IDF + Dense | RAGAS + Qwen2.5 | 0.754 | 0.668 | 0.778 | — |
| Dense | GPT-5.6 Sol | 0.831 | 0.753 | — | 0.912 |
| TF-IDF + Dense | GPT-5.6 Sol | 0.848 | 0.775 | — | 0.921 |
Context changed
Answer changed
Manual judge check
GPT-5.6 Sol aligned much better with the manual judgments than the smaller local judge in this small spot-check.
A simple routing layer beat complexity on the accuracy/speed trade-off.
The project showed that retrieval should be diagnosed before adding heavier components. TF-IDF policy routing substantially cleaned the search space at negligible cost; cross-encoder reranking improved ranking further but added ~400 ms CPU latency. The resulting classical pipeline is preserved as the baseline for the GraphRAG study.