SAB: Startup Agent Benchmark

As of October 2026, SAB compares 14 models from four vendors on 960 research tasks and 101 ranking cohorts.

Claude Opus 5.5 leads the research tasks with 97.2 of 100 points (Claude Fable 5.1 96.8; intervals overlap) and passes 96% of them under the strict rule, at less than a third of Fable 5.1's cost per task. Every model gives a value for some of the facts we removed from the corpus, from 9% to 33% of them. On ranking, all 13 models that completed the cohorts score below a heuristic that orders companies by past headcount growth: their Spearman ρ with realized growth ranges from −0.16 to −0.003, against +0.06 for the heuristic.

97.2

Best research score of 100, Claude Opus 5.5

9–33%

Withheld facts for which a model still gave a value

13 of 13

Models that completed the ranking cohorts and score below the headcount-growth heuristic

$0.003–0.36

List-price cost of one research task, cheapest to most expensive model

Leaderboard

One row per model, ordered by research score.

Hover a row to highlight the model in every chart on the page, and click it to keep the highlight. Click a column header to sort. Scores use the answer keys corrected after the audit (v0.7.1).

Research score: mean points per task over 960 tasks, 80 for answer content and 20 for evidence, with a 95% stratified-bootstrap interval. Rank: 95% bootstrap interval of the rank. Pass: share of tasks with at least 80 points including 10 for evidence, every critical field correct and no nonexistent citation. Utility: from −1 to 1; rewards unknown on withheld facts and penalizes values given for them. Withheld: share of the withheld facts for which the model gave a value. $/task: list-price cost of one research task. Ranking ρ: mean Spearman correlation between the model's ranking of 100 companies and their realized growth, over 101 cohorts.

Strict pass rates order the models differently from mean scores: Grok 4.6, sixth by mean score, is fourth by pass rate.

Research tasks

Ranking tasks

Left: research score with 95% interval (filled circles) and strict pass rate (open diamonds); answering unknown everywhere scores 7.5. Right: Spearman ρ between each model's ranking of 100 companies and their realized growth in web traffic and headcount, averaged over the 90- and 180-day horizons and 101 cohorts, with 95% intervals over cohorts. A random order scores 0 in expectation.

Research

960 tasks in six analyst workflows, half on synthetic companies and half on a frozen snapshot of Mento's company data.

#01

Difficulty by family

Sourcing screens (90.5–100 in both tracks) and real-world product tasks (99–100) are near the ceiling for every model. Averaged over the two tracks, models differ most in market & why-now (53–97) and in competition (60–96). The lowest single scores are in synthetic market & why-now (32–100), real-world competition (28–94) and real-world traction (68–96). In synthetic market tasks, Claude Haiku 4.5 and DeepSeek-V4-Pro answer unknown on most answerable fields.

Mean research score per track and family, 80 tasks per cell.

#02

Cost

At list prices a research task costs from $0.003 (GPT-6 Luna) to $0.359 (Claude Fable 5.1), a factor of more than 100. The Pareto frontier runs through GPT-6 Luna, GPT-5.6 Luna, GPT-6 Sol and Claude Opus 5.5, which scores highest at less than a third of Fable 5.1's cost.

Y axis:

Cost per research task at each provider's public list price, from logged token counts (log scale). The dashed line is the Pareto frontier of the observed means.

#03

Synthetic and real-world tracks

The two tracks order the models similarly but not identically (Spearman ρ = 0.78). DeepSeek-V4-Pro scores 82.4 on synthetic tasks and 91.5 on real-world ones; GPT-6 Luna scores 89.7 and 81.0. GPT-6 Sol scores 1.9 points above its predecessor GPT-5.6 Sol, while GPT-6 Luna scores 5.3 points below GPT-5.6 Luna, entirely on the real-world track (81.0 against 91.7; both score 89.7 on synthetic tasks).

Synthetic-track score against real-world-track score. The dashed line is y = x.

#04

Failures and abstention

Citations of nonexistent source IDs are rare: no model makes one in more than two of its 960 tasks. Summed over models, the most common primary causes of failure are a wrong value in a critical field (528 failed tasks), unknown on an answerable critical field (507) and a value given for a withheld fact (464).

Every model gives a value for some withheld facts. Claude Opus 5.5 (9%) and GPT-6 Astra (10%) do so least often; the three GPT-5.6 models, Grok 4.6 and Claude Sonnet 5 (30–33%) most often. Weaker models also err in the other direction: DeepSeek-V4-Pro, GPT-6 Luna, Claude Haiku 4.5 and DeepSeek-V3.2 answer unknown where the key expects a value on 7–10% of answerable criteria, against 0–2.2% for the five best models.

Failed tasks by primary cause

Withheld facts given a value

Left: share of the 960 research tasks that fail, by primary cause. A failed task receives the first applicable cause in legend order. Insufficient evidence means fewer than 10 evidence points; below score threshold means fewer than 80 points. Right: share of the withheld criteria for which the model gave a value instead of unknown.

#05

Resolved differences

Claude Opus 5.5 averages 0.41 points above Claude Fable 5.1 (95% CI −0.02 to 0.87). Three of the 13 differences between adjacent models are significant after Holm correction, and remain so with correction over all 91 pairs: GPT-5.6 Terra over GPT-5.6 Luna, GPT-5.6 Luna over DeepSeek-V4-Pro and DeepSeek-V3.2 over Claude Haiku 4.5. The other adjacent orderings are not resolved at this sample size, where the minimum detectable difference is about 1.6 points.

Paired differences in research score between models adjacent in the leaderboard, with 95% bootstrap intervals and two-sided sign-flip permutation tests, Holm-corrected over the 13 pairs.

#06

Closed-book control

Run with search returning nothing and fetch refused, DeepSeek-V4-Pro, DeepSeek-V3.2 and Grok 4.6 answer unknown almost everywhere, as instructed, and score 7.56, 7.49 and 7.53, close to the 7.53 of an agent that answers unknown to every field. With the corpus they score 86.9, 83.4 and 94.6, so their scores depend on the documents. Because the instruction forbids prior knowledge, this control does not measure what the models know.

Ranking

101 cohorts of 100 masked companies, ranked without tools and scored against realized growth.

Each ranking task gives the model the masked profiles of 100 privately held companies, with a description and a panel of signals observed at a baseline date between August and December 2025, and asks it to rank them from most to least likely to show the strongest operating momentum over the next 90 and 180 days. The prompt does not define operating momentum. A ranking is scored by its Spearman ρ with a target that weights each company's percentile of web-traffic growth by 0.625 and of headcount growth by 0.375, averaged over both horizons and all cohorts. Grok 4.6 completed 9 of 101 cohorts because its requests timed out at the provider, and is excluded.

#01

Correlation with realized growth

The headcount-growth heuristic, which ranks companies by their headcount growth before the baseline (normally over six months), scores ρ = +0.058. All 13 models score between −0.160 (Claude Sonnet 5) and −0.003 (GPT-5.6 Terra), and for each of them the paired 95% interval of the difference from the heuristic excludes zero. Twelve fall below the 95% range of random rankings.

The comparison depends on how the target is measured. Scored on headcount growth alone, every model's ρ is positive (+0.075 to +0.153) and 10 of 13 have point estimates above the heuristic's +0.105; we did not compute paired intervals for that target. Switch the target to compare.

Target:

Mean Spearman ρ between each model's ranking and the selected target. Intervals and the random-ranking range are shown for the release target, on which the paper's tests are run.

#02

Company size

Every model places larger companies earlier: the correlation between its ranking and baseline web traffic ranges from +0.35 to +0.70. In these cohorts smaller companies grew faster in web traffic, and the target correlates with baseline traffic at −0.24. A post hoc rule that ranks the smallest companies first scores +0.24. Baseline traffic is the denominator of traffic growth, so measurement noise, regression to the mean and real differences in growth could all contribute to this association; we report the rule only as a diagnostic of how the target depends on size.

Controlling for baseline traffic, the partial correlation with the target is positive for 12 of 13 models. Six exceed the heuristic's +0.078, and for four of them the paired 95% interval of the difference excludes zero: GPT-5.6 Terra (+0.135), GPT-6 Sol (+0.132), Claude Fable 5.1 (+0.128) and Claude Opus 5.5 (+0.116). We defined this analysis after seeing the raw results and treat it as exploratory.

Raw and size-controlled ρ

Ranking vs. baseline traffic

Left: open circles show the raw ρ with the target; filled circles show the partial correlation that remains after removing the linear effect of baseline-traffic ranks from both the ranking and the target, with 95% cohort-bootstrap intervals. Right: Spearman ρ between each model's ranking and baseline web traffic; positive values mean larger companies first.

#03

Top of the list

A fund acts on the top of a list. Precision at 10, the share of a ranking's first ten companies that are among the cohort's ten highest-target companies, is 0.07–0.11 for the models, against 0.10 for a random order, 0.14 for the heuristic and 0.31 for the smallest-first rule.

Method

Task-scoped tools under fixed budgets, deterministic grading of answers and cited sources, and credit for unknown where the evidence was removed.

#01

Research tasks

Each task names a target company, asks one question, often with an explicit formula or decision rule, and fixes a JSON response schema in which every answer field also admits unknown. Answers are graded on 4,560 criteria by normalized exact match for strings, an explicit tolerance for numbers or set F1 for lists. The keys mark 1,735 criteria as critical, typically those on the fields that the task's question turns on. Each of the six families has 80 synthetic and 80 real-world tasks.

Family	What the agent must do	Main difficulty
Product & customer	Identify the target among look-alikes; return its category, customer type, verticals and technologies	Same-named company in another vertical (synthetic)
Market & why-now	Select an exact peer set and compute peer medians, or compute a bottom-up market size and name the reasons the market is opening now	Near-miss peers; arithmetic over many documents
Competition	Map direct and indirect alternatives (synthetic); rank candidates with a disclosed similarity formula (real-world)	Partial matches on single factors
Team	Assess founder backgrounds and gaps (synthetic); deduplicate roles and compute headcount growth (real-world)	Namesakes; duplicate roles; missing biographies
Traction & risks	Reconcile dated metrics across providers and apply a stated risk rule	Stale figures from authoritative-looking sources
Sourcing screen	Collect every company that matches criteria and remove those already in the fund's database	Near-miss companies; finding every match

#02

Two tracks

The synthetic track (480 tasks) uses fictitious companies produced by a generator, so no model can know them in advance and every trap is placed deliberately. A synthetic environment holds a median of 49 documents; on average 5.5% of them support a graded answer, and the rest are distractors: SEO pages, directory listings, dead links, paywalled stubs and a same-named company in another vertical. Documents carry publication dates but no staleness labels, and in traction tasks an authoritative-looking figure is five months older than the verified one.

The real-world track (480 tasks) is built from a frozen snapshot of 8,000 company records in Mento's data warehouse, retrieved on 21 August 2026. A real-world environment holds a median of 14 documents, about 10 of them distractors derived from other records; in sourcing environments the distractors are companies that narrowly miss the screening criteria. The two tracks contain 30,529 documents in 960 environments.

#03

Withheld facts

For every fifth task in each family, the generator tries to remove one document and labels a criterion unanswerable if the removal leaves it without the sources it needs. It rejects a removal if an affected value still appears verbatim in another document. Under the corrected keys, 95 tasks contain at least one of the 221 withheld criteria. Withheld criteria are critical, so giving any value for one fails the task; the answer that earns credit is unknown.

#04

Scoring

score = Σ_j w_ja_j + ¼ Σ_j w_ja_je_j − min(20, 10f), floored at 0

Criterion weights w_j sum to 80. Accuracy a_j is 1 for an exact match or a number within tolerance and the F1 score for a list; on a withheld criterion it is 1 for unknown and 0 for any value. Evidence e_j is 1 when the answer cites a set of sources that the key marks as sufficient, or answers unknown on a withheld criterion without a citation. f counts cited IDs that do not exist in the task's corpus. The grader checks that cited IDs exist and include the supporting ones; extra valid citations cost nothing.

utility = Σ_j w_ju_j / 80

Utility lies between −1 and 1. Its credit u_j is +1 for unknown on a withheld criterion, −1 for any value there, +0.2 for unknown on an answerable criterion, 0 for a missing field and 2a_j − 1 otherwise. On an answerable criterion, answering pays more than abstaining only if the expected accuracy exceeds 0.6. An agent that answers unknown to every field and cites nothing scores 7.5 points, with a utility of 0.26.

#05

Harness

Tasks are served through an MCP server whose two tools are scoped to the current task's environment: benchmark_search returns up to 10 results and benchmark_fetch returns one document. A synthetic task allows 8 searches and 12 fetches, a real-world task 12 and 20. A frozen harness fixes the system prompt and task rendering, limits each conversation to 32 model turns and 32 tool calls, and validates the final answer against the JSON schema. Every model was accessed through an API, ran zero-shot at high reasoning effort and was run once; we score the first accepted answer to each task. The ranking tasks contain every input and allow no tool calls.

#06

Statistics and cost

All models answer the same tasks, so comparisons are paired. Intervals are 95% percentile intervals from 10,000 bootstrap replicates that resample tasks within each of the 12 track and family cells, with the same draws for every model; the same replicates give the rank intervals. Pairs of models are compared with a two-sided sign-flip permutation test on per-task score differences, with Holm correction. Ranking intervals resample cohorts, keeping both horizons of a cohort together. Costs use logged token counts at each provider's public list price.

Key audit

The research scores on this page use answer keys corrected after the runs.

After the runs we checked, against its documents and prompt, every criterion that few or no models answered correctly. We found keys that contradict the documents, unstated output conventions, mislabeled withheld facts, a cutoff date with two readings, citation rules that demand redundant documents or accept insufficient ones, and keys that expect 0 or none where the instructions ask for unknown. The corrected keys (v0.7.1) are applied to the saved answers without new runs.

The released keys score every model 4.8 to 7.5 points lower, and the two orders of models correlate at ρ = 0.97. An audit that starts from criteria few models answer correctly finds keys that are too strict and misses keys that are too lenient. Only the citation rules were checked exhaustively, and that check lowered scores.

Effect of each correction applied alone to the released keys, averaged over the 14 models. Together the corrections add 6.6 points and 13.7 percentage points of pass rate.

At Mento

What we delegate to agents, and what investors review.

Even the highest-scoring model, Claude Opus 5.5, fails 41 of 960 tasks under the strict rule, among them 16 of the 80 real-world traction tasks, whose reconciled metrics go into an investment memo. In 16 of the 41 it gives a value for a withheld fact. The two models that answer withheld facts least often still give a value for 9–10% of them, so every answered field has to be traced to a supporting source.

The models' rankings correlate positively with company size and negatively with our growth target. Because the prompt did not state the target, we cannot separate forecasting errors from a different reading of the request.

Mento's agents therefore gather evidence with citations and screen companies against explicit criteria, the sourcing-screen family on which every model scores at least 90.5. Investors check the sources and make the ranking and investment decisions. We did not measure the accuracy or cost of that review.

Limitations

Each model was run once, zero-shot, at one reasoning effort, so the intervals reflect variation across tasks only. Provider endpoints do not guarantee identical checkpoints or defaults.
There is no human-analyst or non-agentic retrieval baseline and no ranking model fitted on the signal panel. Closed-book runs cover three models from two vendors.
The real-world track and the cohorts come from one fund's data, which we cannot release in full.
The key corrections were made after we saw the answers and follow our reading of the prompts.
We do not evaluate investment returns. The ranking target, short-term operating growth, is inversely associated with baseline traffic in these cohorts and is not stated to the model. The models' training data may overlap the outcome period (November 2025 to June 2026), and 2.3% of the masked descriptions still name the company.
The synthetic track may carry generator artifacts, and the research tasks are close to the ceiling for the best models.

Next we add a forward ranking track, a human-analyst baseline, repeated runs and a prompt that states the target.

Cite

Maxim Kalpinmax@mento.vc
Alexander Zhuravlevalex@mento.vc
Dmitry Golomidovdima@mento.vc
Sofya Bilichsonya@mento.vc
Violetta Livi@mento.vc

@misc{kalpin2026sab,
  title        = {{SAB}: Benchmarking {LLM} Agents on Startup Research and Ranking},
  author       = {Kalpin, Maxim and Zhuravlev, Alexander and Golomidov, Dmitry
                  and Bilich, Sofya and Li, Violetta},
  year         = {2026},
  howpublished = {Mento.vc}
}

SAB: benchmarking LLM agents on startup research and ranking

Leaderboard

Research

Difficulty by family

Cost

Synthetic and real-world tracks

Failures and abstention

Resolved differences

Closed-book control

Ranking

Correlation with realized growth

Company size

Top of the list

Method

Research tasks

Two tracks

Withheld facts

Scoring

Harness

Statistics and cost

Key audit

At Mento

Limitations

Cite