Eval timeline
The whole story, in order. Milestones (hollow dots) are real runs from before the eval logger existed — sourced from the git history and docs, on smaller corpora, so they have headline numbers but no per-case detail page (the per-case data was never captured). Logged runs (solid dots) are recorded since the logger, on the 45-case / 4-filing corpus, each with a clickable detail page and a delta vs the previous run.
⚠ These metrics are not one comparable series — each era ran on a different corpus. A smaller eval scored higher (the 6-case 1.00) precisely because it was less trustworthy; growing and hardening the set is what made the numbers honest. Compare within an era, not across.
Milestones · pre-logger (documented)
2026-06-07 · 08f3144 · milestone — no per-case data
v0 — walking skeleton
3 cases (dummy doc)
Mock embedder + in-memory store + mock generator: the pipeline walks end-to-end. No real quality signal yet — by design.
2026-06-12 · 7f9ac66 / 29176bf · milestone — no per-case data
Real-PDF ingestion + page chunking
6 cases (Berkshire 2023)
unpdf ingestion + token-aware per-page chunking. Caught a chunker bug that severed decimals ($96.2 → "$96. 2").
2026-06-13 · 2ea089a / 19fb493 · milestone — no per-case data
Tuning: breadth 5→20, chunks 800→350
6 cases · retrieval 1.00 · answer 1.00
Wider retrieval lifted answer 0.67→0.83; 350-char chunks then fixed the q005 float-dilution miss → 1.00. The 6-case 1.00 was UNTRUSTWORTHY — too small — which is exactly why we grew the set.
2026-06-13 · 69e8dba · milestone — no per-case data
Honest 20-case baseline (local)
20 cases (Berkshire + Meridian) · retrieval 0.82 · answer 0.85
nomic-embed-text + llama3, in-memory. Growing 6→20 (multi-fact + refusal) dropped the score to an honest 0.82/0.85 and surfaced the real misses (q008/q013/q014).
2026-06-16 · b5d3452 / ff72942 · milestone — no per-case data
pgvector store (Neon)
20 cases
PgVectorStore behind the VectorStore seam — serverless-ready, so the index can live off-disk for deploy.
2026-06-17 · d5b8613 / dfea5c7 · milestone — no per-case data
v1 deployed — Jina v3 + Groq 70b
20 cases (Berkshire + Meridian) · retrieval 0.94 · answer 0.95
Hosted stack: Jina v3 + Groq llama-3.3-70b + pgvector. The stronger embedder lifted retrieval 0.82→0.94. Live demo shipped.
2026-06-17 · embedder-comparison.md · milestone — no per-case data
Embedder study (eval-driven selection)
20 cases · retrieval hit@20 · retrieval 0.94
nomic 0.82 · nomic+prefix 0.71 (regressed) · qwen3-0.6b 0.88 · qwen3-8b 0.88 · Jina v3 0.94 (winner). Query/doc asymmetry + reranker shelved as measured negatives. MTEB rank ≠ our-eval.
2026-06-18 · q021–q024 · milestone — no per-case data
Multi-hop probe (24 cases)
24 cases (Berkshire)
Added 4 multi-hop cases (q021–q024). Groq computed same-chunk and one cross-page difference correctly — encouraging, but too few cases to decide on v2.
Logged runs · 45-case corpus (full per-case detail)
2026-06-18 16:21
jina-v3 + deepseek-chat · k=20ab:embedder
retrieval 63% · answer 78%
Deployed config: Jina v3 + DeepSeek, 350-char chunks — the production stack.
2026-06-18 17:15
qwen3-embedding:0.6b + deepseek-chat · k=20ab:embedder
retrieval 48% ▼ -15pp · answer 71% ▼ -7pp
qwen3-0.6b vs jina-v3, same DeepSeek generator and k=20 — isolates the embedder
2026-06-18 18:19
nomic-embed-text + deepseek-chat · k=20ab:embedder
retrieval 55% ▲ +8pp · answer 71% ±0
embedder ladder: nomic-embed-text (768d) — weakest/oldest local model
2026-06-18 21:17
retrieval 57% ▲ +2pp · answer 71% ±0
chunk 800 vs row2 qwen-0.6b@350 — same embedder+generator, isolates chunk size
2026-06-19 08:23
retrieval 55% ▼ -2pp · answer 71% ±0
Jina v3 @ 800 vs row1 Jina@350 — deployed embedder, isolates chunk size (350 was inherited from v1 Berkshire tuning)
2026-06-21 09:56
jina-v3 + deepseek-chat · k=20ab:planner
retrieval 63% ▲ +8pp · answer 78% ▲ +7pp
baseline: no planner (single query vector)
2026-06-21 09:58
jina-v3 + deepseek-chat · k=20ab:planner
retrieval 63% ±0 · answer 80% ▲ +2pp
v2: query decomposition (PLANNER=llm)
2026-06-21 11:16
jina-v3 + deepseek-chat · k=20ab:retrieval
retrieval 63% ▼ -1pp · answer 78% ▼ -3pp
vector-only (baseline)
2026-06-21 11:18
jina-v3 + deepseek-chat · k=20ab:retrieval
retrieval 63% ±0 · answer 78% ±0
v1.5: hybrid BM25+vector (RRF)
2026-06-21 11:21
jina-v3 + deepseek-chat · k=20ab:retrieval
retrieval 70% ▲ +7pp · answer 82% ▲ +4pp
v1.5+v2: hybrid + query decomposition
2026-06-22 02:27
jina-v3 + deepseek-chat · k=20rrf:vw1.0
retrieval 70% ±0 · answer 82% ±0
hybrid+planner, vector weight 1.0
2026-06-22 02:29
jina-v3 + deepseek-chat · k=20rrf:vw1.2
retrieval 73% ▲ +3pp · answer 82% ±0
hybrid+planner, vector weight 1.2
2026-06-22 03:54
jina-v3 + deepseek-chat · k=20chunk:ctx-prefix
retrieval 70% ▼ -3pp · answer 80% ▼ -2pp
contextual prefix + de-noise, hybrid+planner vw1.2
2026-06-22 04:44
jina-v3 + deepseek-chat · k=20rerank:jina-v2
retrieval 73% ▲ +3pp · answer 80% ±0
ctx-prefix + hybrid+planner vw1.2 + Jina reranker (wide 50 then top 20)
2026-06-22 05:20
jina-v3 + deepseek-chat · k=20rerank:jina-rawtext
retrieval 85% ▲ +13pp · answer 87% ▲ +7pp
separated: embed/BM25 contextualized, rerank/gen see RAW text
2026-06-22 06:08
jina-v3 + deepseek-chat · k=20deploy:pg-chunks_v2
retrieval 80% ▼ -5pp · answer 91% ▲ +4pp
pg FTS hybrid + planner + reranker on chunks_v2 (deploy validation)
2026-06-23 05:06
jina-v3 + deepseek-chat · k=20deploy:pg-genfix
retrieval 80% ±0 · answer 98% ▲ +7pp
pg deploy stack + comparison-aware generator prompt (rerun)
2026-06-23 06:04
jina-v3 + deepseek-chat · k=20deploy:pg-judged
retrieval 80% ±0 · answer 98% ±0
deploy stack + comparison-aware generator + LLM judge for freeform (E14)
2026-06-23 09:02
jina-v3 + deepseek-chat · k=20deploy:final
retrieval 83% ▲ +2pp · answer 98% ±0
v2 deploy stack, judge on, q034 gold corrected to income-statement form
2026-06-23 10:06
jina-v3 + deepseek-chat · k=20tbl:inmem
retrieval 75% ▼ -7pp · answer 98% ±0
table-aware chunking, in-memory, full stack
2026-06-25 09:57
jina-v3 + deepseek-chat · k=20tbl:family3
retrieval 65% ▼ -10pp · answer 87% ▼ -11pp
Family 3 multi-representation (LLM row descriptions), in-memory, partial coverage
2026-06-25 10:51
jina-v3 + deepseek-chat · k=20tbl:family3b
retrieval 80% ▲ +15pp · answer 89% ▲ +2pp
Family 3 v2: normal windows + statement-window descriptions, robust describer
2026-06-25 11:58
nomic-embed-text + deepseek-chat · k=20tbl:f3-nomic
retrieval 88% ▲ +7pp · answer 91% ▲ +2pp
Family 3 augment on nomic (no Jina reranker), proxy signal
2026-06-25 17:44
Qwen/Qwen3-Embedding-8B + deepseek-chat · k=20deploy:q8-sf
retrieval 90% ▲ +3pp · answer 89% ▼ -2pp
SiliconFlow Qwen3-Embedding-8B + Qwen3-Reranker-8B + Family 3 descriptions + DeepSeek, pg chunks_q8
2026-06-25 18:01
Qwen/Qwen3-Embedding-8B + deepseek-chat · k=20deploy:q8-sf-xdocfix
retrieval 98% ▲ +7pp · answer 98% ▲ +9pp
qwen8b + Family3 + per-sub-query rerank for cross-doc