Benchmarks
LLM benchmark methodology: how Atlas will measure models
We publish the protocol before the results, so the method cannot be adjusted to fit them. As of today no study has been run, and the site shows no benchmark numbers of its own.
Every Atlas benchmark will name the exact model, provider endpoint or local hardware, quantization, settings and date; use a fixed prompt template with no per-model tuning; publish the task set (or say why it cannot be published), raw outputs where rights allow, and the scripts to rerun it; report absolute counts next to percentages; and say what the result does not show. A study is labelled planned, in progress or published, and only published studies carry numbers.
Evidence policy
Atlas distinguishes four kinds of statement, and the site labels which one you are reading:
| Kind | Meaning | Example |
|---|---|---|
| Measured | We ran it, under a published protocol, and publish the evidence. | None yet. |
| Sourced | A fact read from a primary source we link, with the date checked. | A context length in config.json; a provider's per-token rate. |
| Claimed | Something a creator or provider asserts that we have not checked. | "0/465 refusals" on a model card. |
| Inference | Our reasoning from sourced facts, marked as such. | "Variants of the same architecture need about the same memory." |
We do not copy scores from other leaderboards into an Atlas score. Where a developer's model card reports benchmark results, we may mention them as the developer's claim, with a link. We will not publish a single combined score unless its weighting, version, confidence and missing-data rules are published first; separate dimensions are preferred.
What we plan to measure
- Coding tasks. Pass rate on documented, reproducible unit tests, with failures categorised (wrong answer, syntax error, timeout, refusal, empty output). Licensed or self-written tasks only.
- Instruction adherence and calibration. A small benign set scored by a written rubric, with agreement between two raters reported.
- Latency and throughput. Time to first token and output tokens per second against named endpoints, with sample counts, percentiles and the time window.
- Cost per solved task. Total paid cost divided by tasks solved, always shown with the absolute counts.
- Feature checks. Structured output and tool calling, only where the provider documents support, recorded as pass or fail per case.
Refusal behaviour itself is not one of our tasks: we do not build or publish sets of harmful prompts. Where a refusal happens on a benign task, it is recorded as a failure category.
Planned studies
Planned · version 0.1 (protocol draft)
Coding tasks with unit tests
- Question
- How often does each model produce a patch that passes the hidden unit tests of a small, licensed task set, and does a refusal-reduced variant pass more or fewer tasks than its base model?
- Tasks
- Self-contained Python and TypeScript functions with hidden tests, written for this study or taken from benchmarks whose license allows redistribution. Benign tasks only.
- Metrics
- pass@1 over the full task set; pass@k with k=5 at the stated temperature; failure category (wrong answer, syntax error, timeout, refusal, empty); tokens used per task
- Sample size
- Planned: at least 100 tasks, 5 samples per task per model. Final counts will be published with the results.
- Environment
- Each model through one named provider endpoint or one named local runtime and hardware. Endpoint, version and quantization recorded per run.
- Prompting
- One fixed system prompt and one fixed task template, published with the study. No per-model prompt tuning.
- Randomness
- Fixed temperature and top_p per study; seeds recorded where the API accepts them (some providers ignore seed).
- Exclusions
- Tasks whose solution appears verbatim in public repositories will be removed after a contamination check.
- Limitations
- A small task set measures these tasks, not general coding ability. Provider-side changes (quantization, routing) can change results without notice. Pass rates on generated tests do not measure code quality or security.
- Results
- None. Not run.
Planned · version 0.1 (protocol draft)
Latency and throughput on hosted endpoints
- Question
- What time to first token (TTFT) and output tokens per second does each provider endpoint deliver for a fixed request, at a stated time and region?
- Tasks
- Three fixed prompts (short, medium, long input) with a fixed maximum output.
- Metrics
- TTFT median and 90th percentile; output tokens per second median; error rate; number of samples
- Sample size
- Planned: at least 50 requests per prompt per endpoint, spread over several days and times of day.
- Environment
- One client location, stated with each result. Streaming requests. Client and network timing recorded separately where possible.
- Prompting
- Published with the study.
- Randomness
- Temperature 0 where supported; output length capped so runs are comparable.
- Exclusions
- Requests that fail are counted as errors and excluded from timing statistics, never silently retried.
- Limitations
- Latency depends on load, region and time; a snapshot is not a guarantee. Reasoning models spend tokens before the visible answer, which changes TTFT.
- Results
- None. Not run.
Planned · version 0.1 (protocol draft)
Instruction adherence and calibration
- Question
- Do refusal-reduced variants follow formatting and factual instructions as well as their base models on a small benign set, scored by a written rubric?
- Tasks
- Benign extraction, formatting and factual-recall prompts with known answers.
- Metrics
- rubric score per item; agreement between two human raters; rate of confident wrong answers
- Sample size
- Planned: about 200 items. Final counts published with the results.
- Environment
- Same endpoint and settings for a variant and its base model.
- Prompting
- Published with the study.
- Randomness
- Temperature 0 where supported.
- Exclusions
- Items raters disagree on after discussion are reported separately.
- Limitations
- Small sets cannot rank models in general. Human rubrics carry rater bias; agreement will be reported.
- Results
- None. Not run.
How cost per solved task will be computed
cost_per_solved = Σ (input_tokens × input_rate + output_tokens × output_rate) / 1,000,000 ÷ tasks_solved reported with: tasks_attempted, tasks_solved, total_cost_usd, provider, rate_checked_at
Reasoning tokens count as output where the provider bills them so. Failed and retried requests are included in the cost, because you pay for them.
Reproducibility
- Exact model identifier, revision or commit where available, provider and endpoint, region, and quantization.
- Hardware and runtime version for local runs.
- Sampling settings, seeds where accepted, and the dates of every run.
- Task files, scoring scripts and raw outputs as downloadable files, under a stated license, unless rights or privacy prevent it, in which case we say so.
Conflicts of interest
Atlas has a commercial relationship with MolochAI (see the partner disclosure). When MolochAI is benchmarked, it will be tested with the same protocol, on the same dates, as every other provider in the study, through a normal account, and the result will be published whatever it shows. MolochAI does not review results before publication.
Corrections
If you find an error in a protocol or, later, a result, write to us through the corrections page. Material fixes are logged publicly.