OpenAI-compatible LLM APIs, and where they differ

What “compatible” covers

The OpenAI chat-completions format has become the common interface for hosted and local language models. A compatible server accepts POST /chat/completions with a model and a list of messages, returns a choices array, and can stream the reply as server-sent events. That is enough for the official Python and Node SDKs, and for most tools built on them, to work after you change two settings: the base URL and the key.

Everything else is up to the provider. Two servers can accept the same request and treat half of its parameters differently. The table below shows the differences that matter most, as each provider documents them.

Where providers differ

Two practical consequences. First, a parameter you rely on for correctness, such as stop or response_format, must be tested on each provider, because a silent ignore looks like success. Second, if your code reads reasoning, it needs to look in more than one place.

Probe an endpoint before you depend on it

This script sends two small requests and reports what the endpoint returned: whether your model is listed, the finish reason and usage of a plain request, where reasoning text appears, and whether usage arrives in a stream.

"""Probe an OpenAI-compatible endpoint and report what it actually returns.

Checks: model list, a plain completion, usage, a streamed completion, whether
usage arrives in the stream, and where reasoning text appears. Sends two
small requests, so it costs a few hundred tokens.
"""
import os

from openai import OpenAI, APIStatusError

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"], timeout=60.0, max_retries=1)
MODEL = os.environ["LLM_MODEL"]
report = []


def check(name, fn):
    try:
        report.append((name, "ok", fn()))
    except APIStatusError as e:
        report.append((name, f"HTTP {e.status_code}", e.message[:120]))
    except Exception as e:  # report anything else instead of stopping
        report.append((name, "error", f"{type(e).__name__}: {e}"[:120]))


def models():
    ids = [m.id for m in client.models.list().data]
    found = "listed" if MODEL in ids else "NOT listed"
    return f"{len(ids)} models; {MODEL} {found}"


def plain():
    r = client.chat.completions.create(model=MODEL, messages=[{"role": "user", "content": "Reply with the word ok."}], max_tokens=50)
    msg = r.choices[0].message
    extra = msg.model_extra or {}
    reasoning = [k for k in ("reasoning", "reasoning_content") if extra.get(k)]
    usage = f"{r.usage.prompt_tokens} in / {r.usage.completion_tokens} out" if r.usage else "no usage"
    where = f"reasoning in field {reasoning[0]}" if reasoning else ("reasoning inline in content" if "<think>" in (msg.content or "") else "no separate reasoning")
    return f"finish_reason={r.choices[0].finish_reason}; {usage}; {where}"


def streamed():
    chunks, usage = 0, None
    stream = client.chat.completions.create(
        model=MODEL, messages=[{"role": "user", "content": "Count from one to five."}], max_tokens=60,
        stream=True, stream_options={"include_usage": True},
    )
    with stream:
        for chunk in stream:
            chunks += 1
            usage = chunk.usage or usage
    return f"{chunks} chunks; usage {'present' if usage else 'absent'} in stream"


check("GET /models", models)
check("chat completion", plain)
check("streaming", streamed)

width = max(len(n) for n, _, _ in report)
for name, status, detail in report:
    print(f"{name:<{width}}  {status:<8}  {detail}")

Against the mock server in our repository it printed:

GET /models      ok        1 models; mock-model listed
chat completion  ok        finish_reason=stop; 6 in / 10 out; no separate reasoning
streaming        ok        11 chunks; usage present in stream

Run it with your own LLM_BASE_URL, LLM_API_KEY and LLM_MODEL. The requests are tiny, so the cost is a few hundred tokens.

Parameters to treat with care

• max_tokens. Providers cap output per reply. Send what you need, then check finish_reason for length.
• temperature and top_p. Usually honoured, but defaults differ by provider and by model. Set them explicitly if results must be repeatable.
• seed. Rarely guaranteed with open-weight serving stacks, and ignored by some providers. Do not build tests that depend on identical outputs.
• response_format. Where it is ignored, ask for JSON in the prompt and validate the result with a schema library; retry on parse failure.
• Model-specific options. Some models take extra parameters, such as reasoning_effort, and their cards document pitfalls. The GLM-5.3-Flash card says the parameter defaults to its deepest mode when missing; one variant’s card adds that a small max_tokens at that setting can return an empty reply.

When compatibility is not enough

If you need features outside chat completions, such as embeddings, file uploads, batch jobs or the newer Responses API, check that the provider implements them; many do not. And compatibility says nothing about what is being served. Two endpoints that accept qwen3.8-27b-uncensored and huihui-ai/Huihui-Qwen3.8-27B-abliterated might serve different weights. The API overview separates providers that name the exact repo from those that serve their own builds.