nomic-embed-text + deepseek-chat · k=20
run #26 · 2026-06-25 11:58 · git 16da4bb · tbl:f3-nomic
Retrieval hit@k
88%
Answer accuracy
91%
Cases
45 (40 grounded)
Family 3 augment on nomic (no Jina reranker), proxy signal
Δ vs baseline
Compared to jina-v3 + deepseek-chat · k=20 (2026-06-25 10:51). A retrieval delta with the same generator isolates the embedder/retrieval change.
- ✓ Retrieval solved here (6): q010, q017, q024, q036, q038, q040
- ⚠ Retrieval regressed (3): q022, q023, q028
- ✓ Answer solved here (4): q010, q038, q040, q043
- ⚠ Answer regressed (3): q022, q023, q024
Failures in this run (6)
| Case | Question | Retrieval | Answer | Symptom |
|---|---|---|---|---|
| q013 | What were Berkshire Hathaway's railroad operating earnings in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q022 | How much did Berkshire Hathaway's operating earnings grow from 2021 to 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q023 | How much did Berkshire Hathaway's operating earnings grow from 2021 to 2022? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q024 | How much larger were Berkshire Hathaway's 2023 net earnings than its 2023 operating earnings? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q028 | What was JPMorgan Chase's CET1 capital ratio in 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q043 | Which had the highest net income in 2023: JPMorgan Chase, Microsoft, or Costco? | FAIL | PASS | answered without the gold chunk (parametric leak?) |