Our business model
Who pays us, and what that money cannot buy.
Public benchmarks on this site are free and will stay free. Nobody pays to be included, and nobody can pay to be left out. This page states how we earn money instead, and the specific things that money has no effect on.
How we earn
One line of revenue today.
Live now. Companies buying AI hire us to run private evaluations on their own workflows — their task, their inputs, their success criteria. The buyer pays; the vendors being compared do not.
Planned, not yet live. Two further lines are intended and have not begun. Vendors will be able to hire us to test their products privately. And vendors will be able to buy aggregated market research — which categories buyers in a given segment are evaluating — which will never identify a person or a company.
We will update this page on the day either of those begins, not afterwards. Until then, no vendor has paid Benchology anything.
The firewall
What money cannot buy.
A place in a public benchmark, or removal from one.
The date a result is published, or a delay.
A rerun that other vendors in the same benchmark do not get.
A look at results before publication, beyond checking the facts about their own product.
Any change to a score.
How we choose what to test
The rules are fixed before the numbers are known.
What the first two benchmarks did. Benchmark 01 and Benchmark 02 were not pre-registered. The vendor list, the inclusion criteria and the scoring rules were published with the results, not before the run. Everything needed to check them is public — the methodology, the scoring rules and the raw data, under CC BY 4.0 — but a reader cannot confirm the rules were not adjusted once the numbers were in.
What we commit to from the next benchmark onward. We publish the vendor list, the inclusion criteria and the scoring rules before the run begins, as a timestamped commit in our public repository, so that the rules cannot move after the results are known.
A vendor that meets the published criteria is tested. Vendors cannot opt out.
Disclosure
Who was paying us when we published.
Every report lists which of the tested vendors were paying Benchology customers on the day it was published.
For Benchmark 01 and Benchmark 02, that list is none. No vendor in either study had any commercial relationship with Benchology at publication.
Mistakes
When we get something wrong.
We fix it, and log the date and the change on the corrections page, with the original wording kept alongside so the change can be checked rather than taken on trust.
Schedule a demo
The buyer pays. That is the whole arrangement.
Bring the workflow you want AI to improve. We'll run one evaluation and show you the evidence behind it.
Schedule a demo