Qwen/Qwen3-Embedding-8B + deepseek-chat · k=20
run #27 · 2026-06-25 17:44 · git 16da4bb · deploy:q8-sf
Retrieval hit@k
90%
Answer accuracy
89%
Cases
45 (40 grounded)
SiliconFlow Qwen3-Embedding-8B + Qwen3-Reranker-8B + Family 3 descriptions + DeepSeek, pg chunks_q8
Δ vs baseline
Compared to nomic-embed-text + deepseek-chat · k=20 (2026-06-25 11:58). A retrieval delta with the same generator isolates the embedder/retrieval change.
- ✓ Retrieval solved here (4): q013, q022, q023, q028
- ⚠ Retrieval regressed (3): q038, q040, q042
- ✓ Answer solved here (3): q013, q022, q023
- ⚠ Answer regressed (4): q038, q040, q042, q043
Failures in this run (5)
| Case | Question | Retrieval | Answer | Symptom |
|---|---|---|---|---|
| q024 | How much larger were Berkshire Hathaway's 2023 net earnings than its 2023 operating earnings? | PASS | FAIL | retrieved but generated wrong (generation/arithmetic) |
| q038 | Which company had higher net income in 2023, Microsoft or JPMorgan Chase? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q040 | How much higher was Microsoft's net income than JPMorgan Chase's in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q042 | What was the combined net income of Microsoft and JPMorgan Chase in 2023? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |
| q043 | Which had the highest net income in 2023: JPMorgan Chase, Microsoft, or Costco? | FAIL | FAIL | gold chunk not retrieved → answer ungrounded |