jina-v3 + deepseek-chat · k=20
run #18 · 2026-06-22 05:20 · git 805cec6 · rerank:jina-rawtext
Retrieval hit@k
85%
Answer accuracy
87%
Cases
45 (40 grounded)
separated: embed/BM25 contextualized, rerank/gen see RAW text
Δ vs baseline
Compared to jina-v3 + deepseek-chat · k=20 (2026-06-22 04:44). A retrieval delta with the same generator isolates the embedder/retrieval change.
- ✓ Retrieval solved here (6): q006, q014, q015, q021, q024, q038
- ⚠ Retrieval regressed (1): q017
- ✓ Answer solved here (3): q024, q041, q043
Failures in this run (11)
| Case | Question | Retrieval | Answer | Symptom |
|---|---|---|---|---|
| q013 | What were Berkshire Hathaway's railroad operating earnings in 2023? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q014 | By how much did Berkshire Hathaway's insurance float grow from 2022 to 2023? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q017 | What kind of transportation business does Berkshire Hathaway operate? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q021 | By how much did Berkshire Hathaway's insurance float grow from 2021 to 2023? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q022 | How much did Berkshire Hathaway's operating earnings grow from 2021 to 2023? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q023 | How much did Berkshire Hathaway's operating earnings grow from 2021 to 2022? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q025 | What was JPMorgan Chase's total net revenue in 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q028 | What was JPMorgan Chase's CET1 capital ratio in 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q034 | What were Costco's net sales in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q040 | How much higher was Microsoft's net income than JPMorgan Chase's in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q043 | Which had the highest net income in 2023: JPMorgan Chase, Microsoft, or Costco? | FAIL | PASS | answered without the gold chunk (parametric leak?) |