How to evaluate an LLM for coding

Measured, sourced and claimed

A model’s coding ability is known only for the tasks it was measured on. Before running anything, sort the evidence you already have into three kinds.

A concrete example of the first kind: the HauhauCS GLM-4.7-Flash card advises “If you’re doing agentic coding, use the Balanced variants” and gives no coding benchmark. That is a recommendation, not a measurement.

Refusal-reduced variants need particular care, because the edit that reduces refusals can also change capability. Young (2025) measured GSM8K changes from +1.51 to −18.81 percentage points across abliteration tools and models. Test any variant beside its base, as described in uncensored vs abliterated.

Atlas has not published measurements of its own yet. The protocol we plan to use is on the benchmark methodology page, and the coding model list is a set of candidates to test, not a ranking.

1. Pick tasks from your own work

The task set decides what the result means. Draw it from work your team actually does:

• Closed bug tickets where the fix and a regression test exist in your history.
• Small features or utility functions with a clear specification.
• Refactors where the existing tests define “still works”.
• Data extraction or transformation scripts with known inputs and expected outputs.
• Work in the languages, frameworks and library versions you really use.

Write each task the way a developer would receive it: the specification, the relevant files or signatures, and nothing that hints at the solution. Mix routine and hard tasks. A set made only of hard problems makes every model look alike.

Size affects how you read the score. With N tasks, each task is worth 100/N percentage points of pass rate, so on 20 tasks one extra success moves the score by 5 points. The last section shows how to judge whether a difference is larger than noise.

Keep the set private. Tasks you publish or paste into public trackers can end up in future training data.

2. Score by running tests

Grade outputs by executing them. Chen et al., who introduced HumanEval, evaluated “functional correctness”: generated programs were run against unit tests instead of being compared with a reference text (Chen et al., 2021). Reading code and judging it plausible is a different measurement, and it does not scale.

The tests have to be strong. Liu et al. extended HumanEval’s tests 80-fold to build HumanEval+. Across 26 models, the extra tests caught enough incorrect programs to reduce pass@k by up to 19.3–28.9%, and two models that trailed ChatGPT on HumanEval came out ahead of it on HumanEval+ (Liu et al., 2023). Weak tests inflate scores and can reorder models.

For a function-level task, cover the edge cases the specification mentions: empty input, non-ASCII text, boundaries, invalid input. A parametrized pytest file keeps this compact (pytest docs). Here the model’s answer is saved as solution.py and the tests as test_slugify.py:

import pytest
from solution import slugify

@pytest.mark.parametrize("raw, expected", [
    ("Hello World", "hello-world"),
    ("  Trim  extra   spaces ", "trim-extra-spaces"),
    ("Crème brûlée", "creme-brulee"),
    ("C++ & Rust!", "c-rust"),
    ("already-a-slug", "already-a-slug"),
    ("", ""),
])
def test_slugify(raw, expected):
    assert slugify(raw) == expected

For repository-level tasks, borrow the structure of SWE-bench. Each instance pairs a real GitHub issue with tests: at least one fail-to-pass test that the fix must make pass, and pass-to-pass tests that check existing behavior still works. A task counts as solved only if all of them pass (Jimenez et al., 2023). Closed bug tickets map onto this directly. The regression test written with the fix is your fail-to-pass test, and the rest of the suite is the pass-to-pass set.

Run generated code in isolation. The HumanEval authors note that executing generated programs “poses a security risk” and built a sandbox for it. Use a container or virtual machine with no network access, no credentials and a time limit.

3. Sample more than once: pass@1 and pass@k

The same prompt can produce working code on one try and broken code on the next, so one sample per task is a noisy estimate. Chen et al. define pass@k as the share of problems for which at least one of k samples passes the unit tests. Computing it from exactly k samples has high variance. Instead they draw n ≥ k samples per task (n = 200 and k ≤ 100 in their paper), count the c samples that pass, and use an unbiased estimator:

pass@k = average over tasks of [ 1 − C(n − c, k) / C(n, k) ]

C(a, b) is the binomial coefficient "a choose b".
For k = 1 this reduces to c / n, the share of samples that passed.

Which number to report depends on how the model will be used:

• pass@1 fits interactive use, where a developer takes the first answer.
• pass@k fits workflows that generate several candidates and keep one that passes the tests, such as an agent loop or a CI-checked batch job.

The gap can be wide. In the Codex paper, the model solved 28.8% of HumanEval problems, while repeated sampling with 100 samples per problem solved 70.2%. Report the sampling temperature with every number, since it changes how varied the samples are.

4. Check for contamination

A model that saw a problem and its solution during training can pass without solving anything. HumanEval’s problems were written by hand for this reason: the authors’ training data came largely from GitHub, which already held solutions to many existing problems.

Two benchmark designs handle the risk with dates. LiveCodeBench collects contest problems over time and records each one’s release date, so a model can be scored only on problems published after its training cutoff. Its authors saw one model’s performance drop sharply on problems released after a certain date, which they read as a sign that earlier problems were contaminated. They also suggest that many models that do well on HumanEval may be overfitting to it. SWE-bench can likewise be extended with tasks from pull requests created after a model’s training date.

For your own set:

• Prefer code that was never public, or that was written after the model’s stated training cutoff.
• Perturb a few tasks: rename functions, change constants, reorder requirements. If the pass rate falls sharply on perturbed versions of tasks the model passed, that suggests memorization rather than problem solving.
• Watch for answers that reproduce the original fix’s exact comments or variable names.

5. Cost per solved task

Price per token is not the cost of getting work done. A cheaper model that needs more attempts, or writes longer answers, can cost more per working solution. Compute it from your run logs:

cost of one attempt  = input tokens × input price + output tokens × output price
cost per solved task = total cost of all attempts / number of tasks solved

Count every attempt, including failures and retries, and include reasoning tokens if they are billed as output. Take prices from the provider’s own pricing page and note the date. The token cost calculator and the API pricing overview help with the arithmetic.

For a self-hosted model, the equivalent is hardware and power time per solved task, which is harder to pin down. At minimum, record the GPU, the quantization and the wall-clock time per task.

6. Latency

Measure what the user will feel:

• Time to first token for interactive use, where a developer waits for the response to start.
• Total time per task, including tool calls and retries, for agents and batch jobs.
• Output tokens per second, for comparing serving setups.

Report the median and a high percentile such as p95 rather than a single run, and measure from the location and hardware you will use in production. For local models, quantization and GPU memory change the numbers. GGUF quantization explains the tradeoff.

7. Tool-calling checks

If the model will act as an agent, it has to call tools correctly as well as write code. The Berkeley Function-Calling Leaderboard sorts calls into categories such as simple, multiple and parallel functions, and includes “Function Relevance Detection”: cases where none of the provided functions should be called. It checks calls by parsing them against the function documentation (AST evaluation) and by executing them and comparing the results.

A small version for your own tools:

Score each check separately. A model can be accurate on arguments and still call tools when it should not. The tool-calling API guide covers how to wire tools up.

8. Record what you need to rerun it

A result you cannot rerun is a claim. For each run, store:

• The model identifier, exact revision (the commit hash for downloaded weights), file and quantization.
• The serving software and version, or the provider and the model ID the API reports, plus the date. A hosted model name is not a version guarantee unless the provider says so.
• The system prompt, prompt template, temperature, top-p, maximum output tokens and seed, if the stack supports one.
• The number of samples per task, the harness version, the container image and the test commands.

Then log one row per sample:

run_id,date,model,revision,quantization,provider,task_id,sample,temperature,passed,tests_passed,tests_total,input_tokens,output_tokens,cost_usd,ttft_ms,total_ms,tool_calls_valid,notes

Run the whole set at least twice, on different days, before drawing conclusions. If results differ much between runs, that variance is part of the answer.

Reading your results

Always report a pass rate together with N. A rough standard error for a pass rate p on N tasks, assuming the tasks do not influence each other, is √(p(1 − p) / N). At p = 0.5 and N = 20, that is √0.0125 ≈ 0.11, or about 11 percentage points, so two models 5 points apart on 20 tasks cannot be told apart. Use more tasks, or compare models pair by pair on the same tasks and count where each one wins.

Keep the three kinds of evidence apart in the write-up. A “source” column with values such as measured by us, reported by creator or third-party benchmark stops a card claim from being read as a measurement.

Finally, keep the conclusion as narrow as the test. “Passed X of N of our Python tasks at temperature T, with these settings” is a finding. “Good at coding” is not.