Retrieval Failure Analysis, Part 2
Retrieval explorer
Inspect the frozen five-question run used in “One Retrieval Method Is Not a Diagnosis.”
Results are frozen; nothing is recomputed in the browser.
Chunking held fixed: section-aware · 13 chunks · same five questions
Retrieval varies: Dense · BM25 · RRF · verdict uses top-k 3
Complete 13-chunk rankings recorded.
Experiment results
| Question | Dense | BM25 | RRF |
|---|
Published verdicts use top-k 3. Method scores are not comparable.
Selected configuration
Chunking: section-aware · 13 chunks
Rubric
Verdict
Missing evidence
Top-3 retrieved chunks
Cosine similarity ≠ confidence. Scores are decimals, not percentages.
Ranks 4–13 · below top-3
Compact ranking of the remaining frozen chunks. Required evidence is marked.
View original TechNova policy
Inspect all 13 section-aware chunks
Ordered by source offset. Top-3 for the selected question and method are highlighted.
Frozen experiment provenance
Fixed setup: section-aware · 13 chunks · 5 questions · top-k 3
Dense: sentence-transformers/all-MiniLM-L6-v2 · cosine similarity
BM25: rank_bm25.BM25Okapi · k1=1.5 · b=0.75 · epsilon=0.25 · no stopword list
RRF: Dense + BM25 · k=60 · fusion from full 13-chunk rankings
Source SHA-256:
59d1785463a4…
Full source SHA-256
59d1785463a48b752babc882ac155ef2b9de84ad7cfed32b64aeecabcb68f1e4
Evidence: preregistration e9d116a · implementation ac7be30 · results 3229255