VERBVISION3 Equity Research Benchmark
600 research tasks across twelve parts of the job, from rebuilding a filed balance sheet to writing a priced thesis.
Measuring the work
analysts actually do.
An agent gets a research brief, a set of filings and a step budget. It has to find the right documents, pick the right periods, adjust for what the accounts hide and show its working. VERB scores the whole derivation, not only the final number.
Answers come from filings
Every numeric reference is recomputed from SEC XBRL facts plus the assumptions stated in the task. Nothing is typed in by hand.
One harness for every model
Same prompt, same tools, same 40-step limit. Each model runs through its own provider's native interface, so no one is handicapped by a translation layer.
Graded on the derivation
Numbers are checked against the reference within tolerance. Written reasoning is scored by two judges from different vendors, with a third model settling disputes.
Only real runs
Every score on this page comes from a logged run. Where a model failed, timed out or submitted a placeholder, that is counted and reported.
Fifteen models.
One harness.
Every model attempted the same 14 tasks (7 core, 3 hard, 4 extra hard), once each, with 40 steps. These tasks come from 5 of the 12 categories. We only score tasks that all fifteen models finished, and this first run reached its budget before covering the rest.
- Anthropic
- OpenAI
Score by category
Rubric score in the 5 categories scored so far. The other 7 follow in the next run. Hover a category for what it covers.
Where the points went
Every attempt is marked against its own rubric. Share of available points earned on each part, for the models selected.
Numerical accuracy 30 pts
Source fidelity and provenance 25 pts
Financial and modelling integrity 20 pts
Analytical reasoning 15 pts
Task completion 10 pts
Judges agreed 92% of the time
Two judges from different vendors scored every written criterion. Where they differed by a full point, a third model decided.
Same budget for everyone
Every model had 40 steps. Between 0% and 29% of attempts reached the limit and were scored as submitted.
A first look
One attempt per task cannot separate models that sit close together. Repeated trials on the 528 hidden tasks come next.
Twelve parts
of the job.
Tasks are weighted toward what fills an analyst's week. Two thirds of the bank is harder work. Hard tasks chain two analyses; extra-hard tasks chain three and hide a trap in each, such as quarters that must be backed out of year-to-date cash flows, leases buried in the notes or prior years that were later restated. Full marks need every figure right.
Historical Financial Statements
Reconstruct, extract and reconcile filed statements.
Complex Accounting & Disclosures
Footnotes, policies and their financial consequences.
Business Economics & Segments
How each business earns revenue and profit.
Earnings, Guidance & Transcripts
Reported results against prior guidance and commentary.
Earnings Quality & Normalisation
Reported versus economic earnings.
Capital Structure & Allocation
Financing decisions and capital deployment.
Forecasting & Integrated Models
Driver-based, balanced forecasts.
Valuation & Market Pricing
Enterprise and equity valuation.
Cross-Border & Peer Comparability
Comparing issuers across standards and currencies.
Specialist Sector Research
Sector-specific schedules and conventions.
Investment Theses & Catalysts
Evidence-based theses, risks and catalysts.
End-to-End Analyst Research
Full multi-step research workflows.