# SAB: Benchmarking LLM agents on startup research and ranking

Mento.vc, October 2026. Release `sab-v0.7.0-baa0ceb6c489`, answer keys v0.7.1.
Paper: /sab-bench/sab_paper.pdf

SAB has two parts. Research: 960 tasks in six analyst workflows, half on synthetic
companies and half on a frozen snapshot of Mento's company data; agents search each task's
documents through two tools, and a deterministic grader checks answers and cited source IDs
against private keys. Ranking: 101 cohorts of 100 masked companies, each ranked
without tools and scored by Spearman ρ with the companies' later growth in web traffic and
headcount.

## Research leaderboard

Research score: mean points per task (80 for answer content, 20 for evidence) over 960
tasks, with 95% stratified-bootstrap intervals. Pass: at least 80 points including 10 for evidence,
every critical field correct and no nonexistent citation. Utility rewards `unknown` on withheld
facts and penalizes values given for them (−1 to 1). Ranking ρ: mean Spearman correlation with
realized growth over 101 cohorts; – means the model did not complete the cohorts.

| # | Model | Research score [95% CI] | Rank | Pass % | Synthetic | Real-world | Utility | Withheld facts answered % | $/task | Ranking ρ |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 97.2 [96.9, 97.5] | 1–2 | 96 | 98.7 | 95.8 | 0.97 | 9 | 0.113 | −0.062 |
| 2 | Claude Fable 5.1 | 96.8 [96.3, 97.3] | 1–2 | 94 | 97.8 | 95.9 | 0.96 | 20 | 0.359 | −0.073 |
| 3 | Claude Opus 5 | 96.2 [95.7, 96.6] | 3 | 94 | 97.2 | 95.2 | 0.94 | 18 | 0.180 | −0.113 |
| 4 | GPT-6 Sol | 95.6 [95.0, 96.1] | 4–5 | 90 | 95.3 | 95.9 | 0.95 | 15 | 0.051 | −0.059 |
| 5 | GPT-6 Astra | 95.2 [94.7, 95.6] | 4–6 | 88 | 93.8 | 96.6 | 0.95 | 10 | 0.166 | −0.045 |
| 6 | Grok 4.6 | 94.6 [93.8, 95.3] | 5–7 | 90 | 96.0 | 93.2 | 0.93 | 30 | 0.112 | – |
| 7 | GPT-5.6 Sol | 93.6 [92.6, 94.5] | 6–8 | 87 | 94.5 | 92.7 | 0.93 | 32 | 0.131 | −0.093 |
| 8 | GPT-5.6 Terra | 93.4 [92.5, 94.2] | 7–8 | 85 | 94.7 | 92.0 | 0.93 | 33 | 0.055 | −0.003 |
| 9 | GPT-5.6 Luna | 90.7 [89.7, 91.7] | 9 | 82 | 89.7 | 91.7 | 0.89 | 31 | 0.008 | −0.121 |
| 10 | DeepSeek-V4-Pro | 86.9 [85.9, 87.9] | 10–11 | 76 | 82.4 | 91.5 | 0.87 | 24 | 0.079 | −0.138 |
| 11 | Claude Sonnet 5 | 85.5 [84.2, 86.7] | 11–12 | 75 | 85.9 | 85.0 | 0.80 | 30 | 0.083 | −0.160 |
| 12 | GPT-6 Luna | 85.4 [84.1, 86.6] | 11–12 | 76 | 89.7 | 81.0 | 0.83 | 19 | 0.003 | −0.063 |
| 13 | DeepSeek-V3.2 | 83.4 [82.1, 84.7] | 13 | 75 | 86.5 | 80.3 | 0.81 | 21 | 0.107 | −0.092 |
| 14 | Claude Haiku 4.5 | 79.8 [78.7, 80.9] | 14 | 67 | 81.7 | 77.9 | 0.73 | 21 | 0.065 | −0.098 |

Reference: answering `unknown` everywhere scores 7.5 (pass 2.6%, utility 0.26).

## Research score by track and family

| Model | Synthetic: Product & customer | Synthetic: Market & why-now | Synthetic: Competition | Synthetic: Team | Synthetic: Traction & risks | Synthetic: Sourcing screen | Real-world: Product & customer | Real-world: Market & why-now | Real-world: Competition | Real-world: Team | Real-world: Traction & risks | Real-world: Sourcing screen |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 98.0 | 100.0 | 100.0 | 95.0 | 99.2 | 100.0 | 100.0 | 93.0 | 89.9 | 99.9 | 92.2 | 99.6 |
| Claude Fable 5.1 | 100.0 | 96.5 | 100.0 | 95.1 | 95.1 | 100.0 | 100.0 | 95.1 | 87.9 | 99.9 | 92.7 | 99.6 |
| Claude Opus 5 | 93.0 | 100.0 | 99.8 | 93.9 | 96.3 | 100.0 | 100.0 | 93.2 | 90.3 | 99.8 | 88.4 | 99.6 |
| GPT-6 Sol | 92.1 | 94.1 | 95.3 | 92.9 | 97.1 | 100.0 | 100.0 | 90.5 | 92.9 | 99.9 | 92.3 | 99.6 |
| GPT-6 Astra | 87.1 | 98.6 | 97.2 | 81.2 | 98.5 | 100.0 | 100.0 | 90.5 | 93.0 | 99.8 | 96.3 | 99.6 |
| Grok 4.6 | 99.0 | 89.1 | 98.4 | 93.7 | 95.6 | 100.0 | 98.8 | 88.6 | 94.2 | 94.2 | 83.8 | 99.6 |
| GPT-5.6 Sol | 99.2 | 84.4 | 95.0 | 93.3 | 95.8 | 99.5 | 100.0 | 91.1 | 82.1 | 93.3 | 90.0 | 99.6 |
| GPT-5.6 Terra | 99.5 | 89.8 | 90.9 | 93.0 | 95.7 | 99.1 | 100.0 | 88.9 | 79.7 | 92.8 | 92.4 | 98.4 |
| GPT-5.6 Luna | 98.8 | 66.0 | 90.9 | 88.9 | 94.7 | 98.7 | 100.0 | 88.9 | 74.3 | 94.4 | 93.8 | 98.7 |
| DeepSeek-V4-Pro | 92.3 | 38.6 | 89.1 | 87.3 | 94.9 | 92.0 | 98.8 | 88.2 | 73.6 | 97.4 | 91.7 | 99.1 |
| Claude Sonnet 5 | 79.5 | 59.0 | 91.3 | 94.0 | 93.8 | 97.8 | 99.9 | 74.4 | 52.7 | 95.2 | 89.8 | 98.1 |
| GPT-6 Luna | 95.6 | 70.0 | 92.4 | 84.9 | 95.4 | 100.0 | 100.0 | 80.1 | 28.0 | 96.6 | 91.0 | 90.5 |
| DeepSeek-V3.2 | 94.4 | 54.4 | 91.2 | 87.4 | 92.8 | 98.8 | 98.8 | 86.7 | 34.5 | 92.5 | 72.5 | 96.9 |
| Claude Haiku 4.5 | 94.9 | 31.8 | 88.2 | 80.7 | 94.8 | 100.0 | 99.7 | 74.8 | 31.8 | 96.2 | 67.7 | 97.1 |

## Ranking

| Model | ρ [95% CI] | Gap to heuristic [95% CI] | Size-controlled ρ | P@10 | ρ with baseline traffic |
|---|---|---|---|---|---|
| Claude Opus 5.5 | −0.062 [−0.080, −0.044] | −0.120 [−0.140, −0.101] | +0.116 | 0.11 | +0.62 |
| Claude Fable 5.1 | −0.073 [−0.091, −0.055] | −0.131 [−0.151, −0.111] | +0.128 | 0.10 | +0.68 |
| Claude Opus 5 | −0.113 [−0.132, −0.094] | −0.171 [−0.190, −0.152] | +0.048 | 0.10 | +0.62 |
| GPT-6 Sol | −0.059 [−0.078, −0.040] | −0.117 [−0.138, −0.097] | +0.132 | 0.10 | +0.65 |
| GPT-6 Astra | −0.045 [−0.064, −0.026] | −0.103 [−0.120, −0.086] | +0.090 | 0.11 | +0.50 |
| GPT-5.6 Sol | −0.093 [−0.113, −0.074] | −0.151 [−0.172, −0.132] | +0.060 | 0.11 | +0.58 |
| GPT-5.6 Terra | −0.003 [−0.021, +0.015] | −0.061 [−0.080, −0.041] | +0.135 | 0.11 | +0.48 |
| GPT-5.6 Luna | −0.121 [−0.141, −0.100] | −0.179 [−0.201, −0.156] | +0.072 | 0.09 | +0.70 |
| DeepSeek-V4-Pro | −0.138 [−0.157, −0.120] | −0.196 [−0.217, −0.176] | +0.013 | 0.08 | +0.62 |
| Claude Sonnet 5 | −0.160 [−0.178, −0.141] | −0.218 [−0.240, −0.196] | −0.084 | 0.08 | +0.35 |
| GPT-6 Luna | −0.063 [−0.080, −0.046] | −0.121 [−0.143, −0.099] | +0.086 | 0.10 | +0.54 |
| DeepSeek-V3.2 | −0.092 [−0.111, −0.073] | −0.150 [−0.174, −0.125] | +0.062 | 0.07 | +0.57 |
| Claude Haiku 4.5 | −0.098 [−0.116, −0.078] | −0.155 [−0.178, −0.134] | +0.048 | 0.07 | +0.56 |
| Headcount-growth heuristic | +0.058 [+0.041, +0.075] |  | +0.078 | 0.14 |  |
| Smallest-first rule (post hoc) | +0.242 [+0.224, +0.260] |  |  | 0.31 |  |
| Random ranking, 95% range | [−0.017, +0.016] |  |  | 0.10 |  |

## Key audit

Scores above use answer keys v0.7.1, corrected after an audit of the released keys.
Each row is the mean effect of one correction over the 14 models.

| Correction | Change | Model-task scores changed | Δ score | Δ pass (pp) |
|---|---|---|---|---|
| Competition keys | Similarity recomputed from the tags the profile documents show; any tied candidate accepted; closest score within 0.01; scorecard entries matched by company name (full or without legal suffix) and score (±0.01) | 895 | +1.87 | +4.1 |
| Removed inputs | Synthetic traction labels whose stated rule needs a withheld document relabeled unanswerable (24 criteria) | 159 | +0.13 | -0.1 |
| Derivable withheld facts | Withheld fields that the surviving documents settle relabeled answerable: real-world risk label and provider months, synthetic identity risk, backgrounds and missing capability (41 criteria) | 364 | +0.54 | +1.4 |
| Unstated formats | Risk and capability labels mapped from free text by keywords; provider-month and background formats; spread ratio to two decimals; tolerances applied without binary rounding error | 1,699 | +2.83 | +7.0 |
| Cutoff reading | Real-world traction answers that exclude provider observations after the stated cutoff month also accepted | 175 | +0.35 | +0.7 |
| Citation rules | Each multi-document rule checked against the facts it needs: more lenient for founder facts (team page or both profiles), a domain risk (domain page) and peer membership (profiles only); stricter for peer medians (metric pages), traction decision labels (every document the rule reads) and the closest competitor (every candidate profile) | 2,306 | -0.50 | -1.0 |
| Absent facts | `unknown` accepted where the key expects 0 from zero funding or an empty team record, and for the unnamed `none` lead and 0 years of domain experience | 434 | +0.56 | +1.5 |
| Copied spans | A string answer credited when it contains the expected value and is copied verbatim from a supporting document | 597 | +1.12 | +1.5 |

| Model | Score, corrected keys | Score, released keys | Pass %, corrected | Pass %, released |
|---|---|---|---|---|
| Claude Opus 5.5 | 97.2 | 90.1 | 96 | 78 |
| Claude Fable 5.1 | 96.8 | 90.1 | 94 | 79 |
| Claude Opus 5 | 96.2 | 89.2 | 94 | 76 |
| GPT-6 Sol | 95.6 | 88.2 | 90 | 74 |
| GPT-6 Astra | 95.2 | 89.4 | 88 | 71 |
| Grok 4.6 | 94.6 | 88.4 | 90 | 75 |
| GPT-5.6 Sol | 93.6 | 86.4 | 87 | 73 |
| GPT-5.6 Terra | 93.4 | 86.2 | 85 | 73 |
| GPT-5.6 Luna | 90.7 | 83.4 | 82 | 66 |
| DeepSeek-V4-Pro | 86.9 | 79.8 | 76 | 63 |
| Claude Sonnet 5 | 85.5 | 80.1 | 75 | 68 |
| GPT-6 Luna | 85.4 | 77.8 | 76 | 63 |
| DeepSeek-V3.2 | 83.4 | 77.0 | 75 | 63 |
| Claude Haiku 4.5 | 79.8 | 75.0 | 67 | 61 |