Qwen/Qwen3-Embedding-8B + deepseek-chat · k=20
run #32 · 2026-07-21 01:04 · git be52b65
Retrieval hit@k
98%
Answer accuracy
96%
Cases
45 (40 grounded)
Δ vs baseline
Compared to Qwen/Qwen3-Embedding-8B + deepseek-chat · k=20 (2026-07-20 11:20). A retrieval delta with the same generator isolates the embedder/retrieval change.
- ⚠ Answer regressed (1): q014
Failures in this run (3)
| Case | Question | Retrieval | Answer | Symptom |
|---|---|---|---|---|
| q014 | By how much did Berkshire Hathaway's insurance float grow from 2022 to 2023? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q024 | How much larger were Berkshire Hathaway's 2023 net earnings than its 2023 operating earnings? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q043 | Which had the highest net income in 2023: JPMorgan Chase, Microsoft, or Costco? | FAIL | PASS | answered without the gold chunk (parametric leak?) |