Data & accuracy

How recommendations are made

The ranking method, the competence bar, and why there's no LLM involved.

3 min read

There is no language model in the loop

Answers are computed directly from the catalogue by a deterministic analyst. No model provider is called, nothing is generated. That has three consequences worth knowing:

  • The same question always produces the same answer.
  • No figure can be invented — every number traces to a row in the catalogue.
  • The consultant cannot reason about anything outside the catalogue, which is why it declines off-topic questions instead of improvising.

How a recommendation is built

  1. Your question is parsed into an intent, an objective, a task and a set of constraints.
  2. Constraints filter the catalogue — capability flags, price ceilings, context minimums, licensing.
  3. Remaining models are scored on the benchmarks relevant to your task.
  4. Models without enough benchmark coverage to score are excluded, and the answer says how many were.
  5. The shortlist is ranked, and the runner-up and cheapest options are checked for the trade-off section.

The competence bar

When you ask for the cheapest or fastest model that does something, optimising on price alone would return a model that technically has the capability flag but is far too weak for the work. So candidates must first clear a fit score of 55 on the task before the price sort runs.

If fewer than three models clear the bar, it is dropped and the answer tells you so — a stated assumption is more useful than a silently narrowed one.

What we don't do

We don't run our own evaluations, don't take payment for placement, and don't weight any maker preferentially. The ranking is arithmetic over third-party benchmark data and published prices.