How the web search API benchmark is run

1 · Scope and intended use

What this measures. One query in, one ranked list of ten results out. We measure whether that list contains the source needed to answer the question, how fast it comes back, and what it costs.

What this does not measure. Answer synthesis, content extraction, multi-step agentic retrieval, index freshness, non-English or non-US performance, or anything about the vendor as a business. Every provider is configured to its plainest search endpoint so that the same layer is compared across vendors: no answer APIs, no content add-ons, no deep or multi-step modes, no query rewriting.

Vendor reliability, pricing stability, compliance and procurement posture are out of scope here and are covered in Part B. The two documents use deliberately different provider universes; see section 3.

Who this is for. Engineers selecting a retrieval layer, and anyone auditing our numbers.

2 · Definitions

3 · Universe and inclusion

Seven providers, selected to span the range of index provenance and price in the agent-search market as of the run window: two operators of independent neural or crawled indexes, two aggregators, one browser-company index, one newly launched independent index, and one Google SERP reseller.

Serper appears in this document and not in Part B. It was excluded from the reliability scorecard after scoring at the floor on all five dimensions, on the grounds that a single all-zero row compresses the visual range without adding information. It is retained here because raw Google organic is the reference point against which every other provider's index independence is read, and removing it would weaken the benchmark. This asymmetry is deliberate. Readers should note that excluding an outlier from the scorecard also narrows the spread reported there, which flatters the remaining six.

4 · Constants

Every value below is held fixed across all provider results. The provider is the only variable.

• k (results per query): 10
• Locale: US, English
• Freshness filter: none
• Domain include/exclude lists: none
• Query text: the SimpleQA problem field, verbatim, with no rewriting by us
• Query rewriting by the provider: disabled wherever the provider exposes the toggle (see section 9)
• Content or extraction: not requested from any provider
• Answer synthesis: not requested from any provider
• Sample: n = 1,000, seeded and deterministic
• Run window: [RUN_START] unresolved to [RUN_END] unresolved
• Price verification date: 2026-09-03

5 · Dataset

Source. SimpleQA (OpenAI), distributed as a CSV at https://openaipublic.blob.core.windows.net/simple-evals/simple_qa_test_set.csv (mirror: basicv8vc/SimpleQA on Hugging Face). 4,326 rows. Three columns: problem, answer, and metadata. The metadata field is a stringified dict containing topic, answer_type, and urls.

Why this dataset. Each row carries both a short verified answer string and a list of supporting source URLs. That combination makes it usable as an information retrieval test collection, which is not how it is normally used. It also has known reference answers, so no LLM grader is required.

Sampling. n = 1,000, drawn by stratified sampling across the 10 topics × 5 answer types (50 cells), proportional to the distribution of the full 4,326-row set. Maximum topic-share drift between the sample and the full set is 0.10 percentage points. The draw is seeded; the seed is published with the data.

Exclusions. Rows with an empty or unparseable urls field are excluded from source-match metrics and reported separately. [COUNT_EXCLUDED] unresolved rows were excluded on this basis.

Run volume. 7,000 calls (1,000 questions × 7 providers), of which 6,998 succeeded. The two failures are documented in section 10.

6 · Metric 1 — Exact source rate

6.1 Definition

The share of questions where at least one of the ten returned URLs matches at least one of the question's reference URLs, after canonicalisation. Binary per question, averaged over the sample.

6.2 Canonicalisation

Applied identically to returned URLs and reference URLs before comparison:

• Strip all fragments, including text-fragment anchors (#:~:text=)
• Remove tracking parameters (utm_*, fbclid, gclid, and equivalents)
• Lowercase the host
• Drop www. and m. prefixes
• Collapse Wikipedia language subdomains (en.wikipedia.org, fr.wikipedia.org → wikipedia.org)
• Remove trailing slashes
• Preserve meaningful query parameters, so two distinct pages on the same path are not merged

6.3 How to read it

This asks whether an engine surfaces the specific source a human researcher cited for that fact.

6.4 It is a floor, not a ceiling

SimpleQA's reference list is not exhaustive. A provider that returns a different but perfectly good source scores zero on that question. The true “found a usable source” rate is therefore higher than these numbers for every provider.

We report it anyway because it discriminates. The same measurement at domain level puts all seven providers within 2.2 points of each other (94.8–97.0%), which tells you that everyone can find the right site and that the interesting differences are about which page gets surfaced.

6.5 Uncertainty

95% Wilson score intervals. Providers whose intervals overlap are not distinguishable at this sample size, and the report says so rather than ranking them.

6.6 Grader validation

Matching is deterministic string comparison, so there is no model grader and no grader error to estimate. Residual error comes from canonicalisation, not judgement: a redirect, a mobile variant, or an archived copy can produce a false negative.

We hand-adjudicate a random sample of [N_ADJUDICATED] unresolved non-matching results and report the measured false-negative rate of the canonicalisation procedure with a confidence interval. That figure is published alongside the results, not buried here.

7 · Metric 2 — Latency

7.1 Definition

Mean wall-clock time from request sent to response received, measured with exactly one request in flight globally, over 150 questions (1,050 sequential calls).

7.2 Why it is measured in a separate pass

Timings collected during the main concurrent run are not provider latency.

We verified this with a back-to-back A/B: the same providers measured concurrently and then immediately at concurrency 1 differed by 1.5× to 7.2×. The inflation was not uniform. It tracked the per-provider concurrency we had assigned to respect each vendor's rate limits, and it reordered the ranking — the provider that looked fastest under load was second slowest in isolation, and the reverse.

In-run timings therefore rank our own worker allocation, not vendor speed. They are not used.

7.3 Ordering

Providers are visited round-robin per question in shuffled order, so no provider is systematically measured first and any drift in network conditions is shared evenly.

7.4 Distribution

Means are reported alongside p50 and p95. Tails vary more than means suggest: one provider's p95 is 3.5× its median where the others sit near 1.5×.

7.5 Caveat

Measured from a single location on a single day. These are relative comparisons under identical conditions, not absolute figures reproducible from a different network.

8 · Metric 3 — Price

8.1 Definition

Vendor list price in USD per 1,000 search requests, verified 2026-09-03. All providers in this benchmark bill per request rather than per token.

8.2 Validation against observed spend

Where a provider exposes billing data, list price is checked against reality:

• Exa reports costDollars on every call. Measured across all 1,000 calls: $7.00 per 1,000, matching its advertised rate exactly.
• Linkup was checked by differencing its credit balance before and after. Observed $5.63 per 1,000 against an advertised $5.00. The gap is our own retry behaviour — retried requests are billed but recorded once — not a pricing discrepancy.

Since announced prices held up where they could be checked, announced prices are what we publish.

8.3 Figures of lower confidence

Any rate that could not be read from a live vendor pricing page is marked with ⚠ in every table where it appears, and the reason is stated in the footnote. Currently: Serper, whose public pricing page was not reachable and whose credit packs sit behind signup, so its rate comes from third-party sources. It is also the figure that most changes the comparison, since it would make Serper the cheapest per call by a wide margin. Treat it as indicative and verify against your own account.

8.4 Commitment-dependent pricing

Some vendors price by volume. Where a range exists, the pay-as-you-go rate is used for the headline figure and the range is given in the footnote.

9 · Per-provider configuration

All settings are provider defaults except those listed. Every row links to the vendor's own documentation for the endpoint used.

Two settings deserve explanation

autocorrect=false on Serper: Google's spell correction would silently rewrite the query, and we would be measuring their rewriter rather than their index.

useAutoprompt=false on Exa: query rewriting is part of what Exa sells, so disabling it is a choice rather than a neutral default. Turning it off measures raw index and ranking; leaving it on measures the product. We report the off configuration as the headline and the delta separately for providers that expose the toggle.

10 · Error and retry handling

Recoverable errors are recorded and remain in published results. These include malformed responses and results the provider returned but could not be parsed.

Transport errors (429, 5xx, connection timeouts) are retried up to [N_RETRIES] unresolved times with exponential backoff. Retried requests are billed by the vendor but counted once in our results, which is the source of the Linkup billing discrepancy noted in section 8.2.

Hard failures are reported rather than dropped. Two You.com calls failed on a documented 50-word query limit. This is a real constraint on long questions, not measurement noise, and those two rows are scored as misses for You.com rather than excluded.

Success rate: 6,998 of 7,000 calls.

11 · Limitations

Stated here in full. Nothing in this list is softened elsewhere on the site.

• The gold URL list is not exhaustive so every retrieval number is a floor.
• SimpleQA was adversarially collected against a frontier model so questions skew long-tail. This measures difficult-fact retrieval, not head queries.
• SimpleQA answers are static by design so nothing here speaks to index freshness — a dimension on which providers are known to diverge sharply from their static-fact performance.
• SimpleQA is public and well-known so it may appear in providers’ own tuning or evaluation sets. We do not filter for this and cannot detect it.
• English-only, US locale, k = 10, single geographic vantage point.
• Latency tails vary more than the means suggest.
• Prices are list prices as of the verification date and change without notice.
• Sampling error At n = 1,000, a proportion near 50% carries a 95% margin of roughly ±3.1 points. Running the full 4,326 rows would tighten this to roughly ±1.5.

12 · Reproducibility and data availability

Every provider response and every intermediate judgement is stored, so scoring rules can be changed and the whole study re-scored offline without re-querying any API.

Published with the results:

• The seeded sample (question IDs)
• Per-provider raw responses
• The canonicalisation implementation
• Per-question match results for both metrics
• Per-call latency and billed cost

The provider configuration is frozen in code. Anyone re-running this should expect drift: we re-ran 10% of the sample at +24h to measure it, and report the observed drift rate.

What has actually shipped, file by file, is on the results page.

13 · Version and revision log

Version

[VERSION] unresolved

Run window

[RUN_START] unresolved – [RUN_END] unresolved

Price verification date

2026-09-03

Published

[PUBLISH_DATE] unresolved

Appendix · Notes for implementation

Content and format requirements for anyone building the UI. No styling guidance. Published because they constrain how the results page is allowed to render these figures.

1 · Scope and epistemic status

Read this section before the scores. This is not a measurement. It is a structured review of public documents, and it should be read with different expectations than a benchmark.

What this assesses: what each vendor has publicly committed to, disclosed, or made verifiable, as of the stated verification date.

What this does not assess: whether a vendor will survive, whether its API will still exist in a year, whether its compliance controls actually work, or whether its retrieval is any good. The last of those is Part A.

Three consequences follow, and we state them rather than hoping they go unnoticed:

1. No confidence intervals apply. These are ordinal judgements against published anchors, not estimates of a population parameter. A score of 3 is not 3.0 ± anything.
2. A low score means poor disclosure, not poor practice. A vendor may hold certifications it has not published, or honour commitments it has never written down. We score the public record because that is what a buyer can act on before signing.
3. The scores are reproducible but not automatic. Another reviewer applying these anchors to the same artifacts should reach the same score. Where judgement was required, the section for that dimension says so.

Who this is for. Security, legal, finance and procurement reviewers who can veto a purchase an engineer has already made.

2 · Definitions

2.1 Evidence states

Every cell carries one of three states. The distinction between the second and third is load-bearing.

2.2 Source grades

Every observed cell also carries a source grade.

A dimension whose score rests mainly on vendor-claimed evidence is weaker than one resting on directly verifiable evidence, even at the same numeric score. The UI must expose this; see the appendix.

2.3 Other terms

Adverse change

A price increase, free-tier reduction or removal, or the introduction of a restriction that did not previously apply.

Grandfathering

Existing customers retained on prior terms through the end of their term after a change.

Effective-price drift

Per-query cost moving without an announced price change, via surcharge stacking, unit redefinition, or depth multipliers.

Trust center

A vendor-hosted page purporting to serve compliance artifacts. Its existence is not the same as a retrievable report.

3 · Universe and inclusion

Six providers: Exa, Tavily, Linkup, You.com, Brave Search API, Keenable.

Serper was scored and then excluded. It scored at the floor on all five dimensions. A single all-zero row compresses the visual range of a five-axis chart without adding information, so it was removed from the scorecard.

Both halves of that decision matter, and we state both. Removing an outlier is defensible on presentation grounds. It also narrows the spread across the remaining six, which flatters them. Serper's full scored row is published in the appendix data file so the decision can be checked.

Serper remains in Part A, where raw Google organic serves as the reference point for index independence. The two documents use different universes deliberately.

4 · Scoring model

Scale. Five discrete levels, 0 to 4, per dimension.

Why discrete. Continuous scores imply a precision that document review does not have. Five levels are enough to separate a vendor that publishes nothing from one that publishes everything, without inviting arguments about whether something is a 2.7 or a 2.9.

Why no weighted composite. A single blended number requires weights, and any weighting we chose would encode our judgement about what matters to a buyer we have never met. Weighted composites in adjacent fields — security ratings in particular — have drawn sustained criticism for exactly this: opaque weights and unfalsifiable aggregate scores. We publish five axes and let readers weight them.

Why a pentagon. Five axes render as a shape, and shape is the finding. A vendor strong on compliance and weak on independence looks different from the reverse at a glance, where two numbers on a list do not.

Direction. All five axes are oriented so that higher is better. This required inverting one dimension: what a buyer experiences as dependency risk is scored here as supply independence, so that a larger shape is never ambiguous.

No composite score is published. No weighted composite, total, or overall rank is published. Any weighting would encode our judgement about a buyer we have not met.

5 · Company-age normalisation

A two-week-old company and a five-year-old company with identical thin disclosure are not the same signal.

Rule. Company age at the verification date is recorded for every vendor and rendered adjacent to its scores. It is not a sixth axis, because it is not something a vendor can be good or bad at — it is a modifier on how thin disclosure should be read.

Rule. Where a dimension requires historical evidence that a vendor has not existed long enough to generate, the cell is scored insufficient history rather than 0, and rendered distinctly. Scoring “no price history” as zero would be a category error.

At the current verification date this applies to Keenable's price stability score only.

The published scorecard still carries TODO_VERIFY_FOUNDING_DATE for four vendors, and records as a publication blocker that “the age normalisation rule cannot be applied without them”.

6 · Missing-data policy

Observed-absent scores lower than not-disclosed. A vendor confirmed to lack an SLA is worse than one whose SLA status we could not determine. Collapsing the two would let opacity score better than honesty.

The residual bias is stated rather than solved. This scorecard still systematically favours vendors with the resources to maintain a trust center, a status page and a documentation site. That correlates with size and funding. A small vendor with excellent internal practice and a thin website will score low here, and that is a real limitation of the method, not an artifact we have corrected for.

Non-disclosure is never inferred into a negative. We do not reason that a vendor which does not publish a SOC 2 report probably lacks one.

7 · Dimension — Price stability

What it signals. How much a buyer's cost basis has moved, and how existing customers were treated when it did.

Artifacts checked. Live pricing page; Internet Archive captures of the pricing page across the 24-month window; vendor changelog and blog announcements of pricing changes; billing documentation; terms covering credit expiry and minimum purchase.

Window. Trailing 24 months from the verification date.

Anchors

Known failure mode. Third-party pricing trackers lag and contradict each other in this category. Tracker figures locate a change, never establish one.

8 · Dimension — Supply independence

What it signals. Whether the vendor can be cut off or repriced by an upstream it does not control.

Artifacts checked. Vendor statements on index provenance; presence of a named crawler user-agent in public AI-crawler registries and in major publishers’ robots.txt; disclosed dependencies on third-party LLMs for synthesis; third-party technical analysis of result composition; active litigation involving the vendor or its upstream.

Anchors

Known failure mode. Crawler registries are verified-bot lists, not censuses. Presence of a named crawler is strong evidence; absence is weak evidence of anything.

Planned strengthening. Cross-provider result-set overlap computed offline from stored retrieval-benchmark responses would move several cells from vendor_claimed to directly_verifiable. Not yet run. When it is, this section and the affected scores are versioned.

Direction. Inverted from buyer-facing 'dependency risk' so that higher is better on every axis.

9 · Dimension — Documentation and commitment

What it signals. Whether the vendor has made the operational commitments a production integration depends on.

Method. Count of eight artifacts, present or absent. The most mechanical of the five dimensions. Judgement is limited to whether an artifact meets the stated bar.

Artifacts checked

1. Public changelog with an entry in the last 90 days
2. Public status page with retained incident history
3. Published deprecation or sunset policy stating a notice period
4. Published versioning policy gating breaking changes
5. Published rate limits and documented 429 behaviour
6. SLA stating an uptime percentage with service credits, at any tier
7. SDK released in the last 90 days for at least one major language
8. No known drift between documentation and observed behaviour

The raw count is published alongside the score, since it is more informative than the mapped value.

Anchors

Known failure mode. The 90-day recency window penalises a mature, stable product with nothing to announce. Raw count is published so readers can disagree.

10 · Dimension — Compliance

What it signals. Whether the vendor clears the first filter of a standard vendor security review.

Artifacts checked. SOC 2 Type II; ISO 27001; live trust center and whether it serves a retrievable report; DPA availability; zero data retention and whether it is default, on request, or enterprise-only; HIPAA with BAA; data residency options; published sub-processor list.

Anchors

Known failure mode. Least objective of the five. Several scores rest on trust-center existence rather than a retrieved report. Treat as an upper bound.

Read this before citing a compliance score. This is the least objective of the five dimensions. Several scores rest on trust-center existence rather than a retrieved report, which is the difference between a claim and an artifact. Until the verification pass in section 12 is complete, treat compliance scores as an upper bound, and read the source grade on each cell rather than the number alone.

Not assessed. Whether controls function. A SOC 2 Type II report is evidence controls were audited, not that they work for a given use case, and it does not remove sub-processor status or DPA obligations.

11 · Dimension — Procurement maturity

What it signals. Whether an organisation with a procurement function can actually buy this, and how much friction it will meet.

Artifacts checked. MSA availability; DPA availability; invoicing, purchase orders and W-9 versus card-only; listing on AWS, Azure, GCP or IBM Cloud marketplaces; existence of an enterprise tier and what it unlocks; named reference customers and whether they are substantiated case studies or logos; enterprise access controls (SSO, RBAC, IP allowlisting); stated support for security review.

Anchors

Known failure mode. Logo walls counted as unsubstantiated unless corroborated by a case study, joint announcement, or marketplace listing. May understate vendors with enterprise customers under NDA.

12 · Verification protocol

Standing rules.

• A trust center's existence is not a retrieved report. Cells resting on trust-center existence are graded vendor-claimed until an artifact is retrieved.
• A third-party tracker may be used to locate a change, never to establish one.
• Where the vendor's own primary source and a tracker disagree, the primary source wins and the conflict is logged in section 13.

Verification status is published per cell, not per vendor. The data file records, for each cell: the evidence state, the source grade, the source URL, and the retrieval date.

Re-verification cadence. Every 90 days, or immediately on a known event (acquisition, pricing announcement, certification). Price stability and compliance decay fastest.

Expiration. Scores carry a validity date. Past it, the scorecard is marked stale in the UI rather than silently served as current. The current scorecard was verified 2026-09-03 and is valid until 2026-12-02.

13 · Known source conflicts

Conflicts are reported, not resolved by preference. This section is part of the findings, not housekeeping.

Rendered from the published artifact, not from the source document. scorecard.json records revisions dated 2026-09-08 that rewrote three of these entries after a fact check — the source markdown for this section predates them and describes the You.com change in the opposite direction to the record. The artifact is the authority here, and its revision log is reproduced on the results page.

14 · Right of reply

Before publication, each vendor receives the rows concerning it, the anchors used, and the evidence cited, with [N] unresolved working days to respond.

• Factual corrections with a citable artifact are applied and logged in section 16.
• Disagreements about interpretation are published alongside the score, not merged into it.
• No response is recorded as no response, and does not change a score.
• We do not offer to change a score in exchange for anything, including access, data, or advertising.

Nothing in this process makes a vendor a reviewer of its own row.

This has not been carried out for the published edition. The artifact records right_of_reply.conducted: false, and “Right of reply has not been conducted” is one of the 11 publication blockers it carries. The paragraph above is the policy, not a description of what happened.

15 · Limitations

Stated here in full. Nothing in this list is softened elsewhere on the site.

• Disclosure is not practice. A vendor may honour commitments it has not published, or publish commitments it does not honour.
• The scorecard favours vendors with resources to maintain public documentation surfaces, which correlates with size and funding.
• Compliance is the weakest dimension and currently rests substantially on vendor-claimed evidence.
• Supply independence rests partly on registries that are not censuses, so absence of a crawler listing proves little.
• Six vendors is a small field, and excluding Serper narrows the observed spread.
• The window is 24 months. A vendor with a poor record 30 months ago scores clean here.
• These scores expire. In a category where a free tier was withdrawn and three vendors repriced within one year, a scorecard older than a quarter should not be relied on.
• This is not a prediction. Nothing here estimates the probability that any vendor fails, is acquired, or discontinues its API.

16 · Version, verification date and revision log

Version

[VERSION] unresolved

Verification date

2026-09-03

Valid until

2026-12-02

Published

[PUBLISH_DATE] unresolved

The artifact's own revision log carries further dated entries against individual vendors; it ships in scorecard.json and is summarised on the results page.

Appendix · Notes for implementation

Content and format requirements for anyone building the UI. No styling guidance. Published because they constrain how the scorecard is allowed to render.