jina-v3 + deepseek-chat · k=20
run #25 · 2026-06-25 10:51 · git 16da4bb · tbl:family3b
Retrieval hit@k
80%
Answer accuracy
89%
Cases
45 (40 grounded)
Family 3 v2: normal windows + statement-window descriptions, robust describer
Δ vs baseline
Compared to jina-v3 + deepseek-chat · k=20 (2026-06-25 09:57). A retrieval delta with the same generator isolates the embedder/retrieval change.
- ✓ Retrieval solved here (6): q006, q008, q009, q023, q028, q042
- ✓ Answer solved here (2): q023, q042
- ⚠ Answer regressed (1): q010
Failures in this run (8)
| Case | Question | Retrieval | Answer | Symptom |
|---|---|---|---|---|
| q010 | What were Berkshire Hathaway's after-tax investment gains in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q013 | What were Berkshire Hathaway's railroad operating earnings in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q017 | What kind of transportation business does Berkshire Hathaway operate? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q024 | How much larger were Berkshire Hathaway's 2023 net earnings than its 2023 operating earnings? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q036 | What was Costco's net income in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q038 | Which company had higher net income in 2023, Microsoft or JPMorgan Chase? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q040 | How much higher was Microsoft's net income than JPMorgan Chase's in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q043 | Which had the highest net income in 2023: JPMorgan Chase, Microsoft, or Costco? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |