What we got wrong, and when we fixed it.

2026-10-26 · Benchmarks 01 and 02 — web search APIs and GTM data providers

Web search API and GTM data benchmarks, September 2026: seven statements corrected

The search API summary said the best retriever, Exa, was “fifth of seven on speed”. That was its rank in our first, concurrent latency pass; in the sequential pass the page reports, it is fourth.

You.com's exact-source rate was published as 89.1% (95% interval 87.0–90.9%), computed over the 998 calls that succeeded. Our method counts hard failures rather than dropping them, so the rate is 88.9% over all 1,000 questions (86.8–90.7%). The ranking does not change. Recomputed, You.com's interval (86.8–90.7%) overlaps Linkup's (82.4–86.9%), so those two providers are no longer distinguishable at this sample size; the results table now says so beside its grouping.

The search API latency section said that measuring one request at a time moved the numbers “by 1.5× to 7.2×”, that the change tracked the concurrency we had assigned each provider, and that “the provider that looked fastest under load was second slowest in isolation, and the reverse.” The published latency file does not support these statements. Each provider's mean in the sequential pass is 1.05× to 1.49× its mean in the concurrent pass, and Keenable is fastest in both. The order changed in the middle: Tavily went from fourth fastest to slowest. The methodology made the same claims and is corrected the same way. The means are over successful calls: Brave and You.com each had one failed call of 150 in the sequential pass.

The search API latency section said “Serper's p95 is 4,911 ms, about 3.5× its median, where the other six sit near 1.5×.” Serper's figures hold (4,911 ms, 3.5×), but the other six range from 1.36× to 2.39×, highest for Keenable. The page now gives that range.

The latency view of the web search chart, on the homepage and the study page, said it showed mean response time “across the same 1,000 tasks.” Latency was measured in a separate pass, one request at a time, over 150 questions per provider. The chart now says so.

The GTM summary said Apollo “returned more valid rows” on the Singapore marketing and Austin founder sets; Apollo had the higher valid-row rate there, not more valid rows.

The GTM summary also said both vendors “sit near 92% on employer fields, meaning about one row in twelve carries a title or company the person's own profile contradicts.” The published file counts title and company agreement separately, between 91.5% and 92.6% for the two vendors. It does not show how often a row has either one wrong, so the “one row in twelve” statement has been removed.

The raw data files were not modified. Both study pages carry a dated notice of these corrections, and the search methodology's revision table records the original and corrected figures.

2026-10-22 · Company policy

Company policy update

Benchology does not accept vendor-funded work and does not share non-public results with vendors before publication. Current company pages now reflect this policy. Earlier study artifacts retain their original policy and process records; benchmark results are unchanged.

Previously, our business model page said: “Vendors will be able to hire us to test their products privately. And vendors will be able to buy aggregated market research”, listed as planned and not yet begun. The homepage said: “Vendors can hire us for private testing”. Our page for data providers said: “each vendor receives the rows concerning it, and the evidence cited, before publication.”

2026-09-16 · Benchmark 01 — web search APIs

Three methodology claims corrected against the run database

§10 said: “Two You.com calls failed on a documented 50-word query limit. This is a real constraint on long questions, not measurement noise.” The run count and the vendor were right, but only one of the two failures was the word limit. The other was a 504 network error that consumed 19.4 seconds and exhausted all three retries — which is measurement noise. Both remain scored as misses, because a failure to return a result is a result, but the page now distinguishes the floor the study imposed from the one the product did.

§9 claimed all settings were provider defaults except those listed. Four were neither default nor listed: country=us and search_lang=en to Brave, gl=us and hl=en to Serper, topic=general to Tavily, and includeImages=false to Linkup. All four narrow results to US English. Retrieval rates are comparable only if the requests are, so a reader reproducing the run from §9 alone would have sent different queries and had no way to tell whose numbers were wrong.

§9 did not disclose that Keenable ran at its “pro” tier. All 400 sampled Keenable responses in the published run report it.

2026-09-12 · Benchmark 02 — GTM data providers

Published: Apollo against Clay across five ICPs

First publication. No corrections to date.

Note added 2026-10-26: two statements on this study's page were corrected on 2026-10-26. See that entry above.

2026-09-03 · Benchmark 01 — web search APIs

Published: seven web search APIs, with Serper excluded from the reliability chart

First publication. Scores were verified the same day they were published.

Recorded at publication rather than corrected afterwards: Serper scored at the floor on all five reliability dimensions. A single all-zero row compresses the visual range of a five-axis chart without adding information, so it was removed from that chart — not from the study. The full scored row is retained in scorecard.json so the decision can be checked, and Serper's retrieval accuracy and latency remain in the results.

Two caveats were published with it. Removing an outlier also narrows the spread across the remaining six, which flatters them. And Serper's price carries a low-confidence flag, because its pricing page was unreachable and its credit packs sit behind signup.