qwen3-embedding:0.6b + deepseek-chat · k=20
run #2 · 2026-06-18 17:15 · git 789d976 · ab:embedder
Retrieval hit@k
48%
Answer accuracy
71%
Cases
45 (40 grounded)
qwen3-0.6b vs jina-v3, same DeepSeek generator and k=20 — isolates the embedder
Δ vs baseline
Compared to jina-v3 + deepseek-chat · k=20 (2026-06-18 16:21). A retrieval delta with the same generator isolates the embedder/retrieval change.
- ✓ Retrieval solved here (1): q008
- ⚠ Retrieval regressed (7): q013, q014, q021, q026, q032, q033, q037
- ✓ Answer solved here (1): q038
- ⚠ Answer regressed (4): q013, q014, q021, q039
Failures in this run (23)
| Case | Question | Retrieval | Answer | Symptom |
|---|---|---|---|---|
| q013 | What were Berkshire Hathaway's railroad operating earnings in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q014 | By how much did Berkshire Hathaway's insurance float grow from 2022 to 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q021 | By how much did Berkshire Hathaway's insurance float grow from 2021 to 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q022 | How much did Berkshire Hathaway's operating earnings grow from 2021 to 2023? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q023 | How much did Berkshire Hathaway's operating earnings grow from 2021 to 2022? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q025 | What was JPMorgan Chase's total net revenue in 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q026 | What was JPMorgan Chase's net income in 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q027 | What were JPMorgan Chase's total assets at the end of 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q028 | What was JPMorgan Chase's CET1 capital ratio in 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q029 | What were JPMorgan Chase's cash dividends declared per share in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q031 | What was Microsoft's net income in fiscal 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q032 | What was Microsoft's operating income in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q033 | What was Microsoft's diluted earnings per share in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q034 | What were Costco's net sales in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q035 | What was Costco's total revenue in fiscal 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q036 | What was Costco's net income in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q037 | How much did Costco earn from membership fees in fiscal 2023? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q038 | Which company had higher net income in 2023, Microsoft or JPMorgan Chase? | FAIL | PASS | answered without the gold chunk (parametric leak?) |
| q039 | Which company had higher total revenue in 2023, Costco or Microsoft? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q040 | How much higher was Microsoft's net income than JPMorgan Chase's in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q041 | How much greater was Costco's total revenue than Microsoft's in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q042 | What was the combined net income of Microsoft and JPMorgan Chase in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q043 | Which had the highest net income in 2023: JPMorgan Chase, Microsoft, or Costco? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |