Methodology
What counts as evidence.
These rules apply to every benchmark we publish. Anything specific to one study — its sample, its instruments, its limitations — lives with that study, and each one is linked at the foot of this page.
Vocabulary
Four things are recorded about every cell.
A score on its own is an opinion. Each cell in a scorecard therefore carries what we found, how strong it is, whether a number applies, and whether we retrieved it ourselves. The definitions below are read directly from the published scorecard.json, so the wording here and the values in the file a reader downloads are the same text.
| Value | Means |
|---|---|
observed | The artifact was found and is present. |
observed_absent | We checked where it would be and confirmed it is not there. |
not_disclosed | Could not be determined from public sources. Not the same as confirmed missing. |
| Value | Means |
|---|---|
directly_verifiable | We retrieved the artifact ourselves and it is independently checkable. |
third_party_reported | Sourced from a tracker, press report, or analysis we did not produce. |
vendor_claimed | Asserted by the vendor with no independently checkable artifact behind it. |
| Value | Means |
|---|---|
scored | A numeric score applies. |
insufficient_history | The vendor has not existed long enough to generate the evidence this dimension requires. score is null, never 0. |
| Value | Means |
|---|---|
verified | Artifact retrieved by us on retrieved_at. |
pending | Not yet independently retrieved. Score provisional. |
The scale
0–4, ordinal, higher is better.
The scale is ordinal, so the distance between 2 and 3 is not claimed to equal the distance between 3 and 4. Every dimension publishes a written anchor for each value, and a score is an assertion that the anchor was met — not a judgement call recorded as a number.
No composite score, and no overall rank. No weighted composite, total, or overall rank is published. Any weighting would encode our judgement about a buyer we have not met.
Missing data
Absence is not scored the same as silence.
Collapsing the two would let opacity score better than honesty. A vendor that documents a weakness therefore scores above a vendor that documents nothing, which is the opposite of what a naive scoring rule produces.
| Confirmed absent scores below undisclosed | Yes |
|---|---|
| Inference from non-disclosure | Not permitted. We do not treat silence as evidence of a practice. |
| Known residual bias | Favours vendors with resources to maintain public documentation surfaces, which correlates with size and funding. |
Publication and corrections
What a vendor can expect.
We do not share non-public benchmark results with vendors before publication. After publication, vendors can submit factual corrections with supporting sources. Confirmed errors are corrected and logged.
Policy updated 22 October 2026. Earlier study files retain the policy and process status recorded for those editions. This update does not change their results or imply a review took place.
Per-study methodology
How each benchmark applies these rules.
- GTM data providers methodology — the sample, the instruments, the per-dimension anchors and the limitations for gtm data providers.
- Web search APIs methodology — the sample, the instruments, the per-dimension anchors and the limitations for web search apis.
Every published result also ships the raw data behind it. If a number here is wrong, it should be possible to prove that from the files rather than to argue about it.
Custom benchmarks
Run the same test on your own ICP.
Bring your ICP and the data providers you use or are considering. We scope a custom evaluation around your decision and show you the evidence: who each provider finds, who your current source misses, and how accurate the records are.