[{"data":1,"prerenderedAt":78},["ShallowReactive",2],{"guides-tool-calling-llm-api":3},{"collection":4,"slug":5,"path":6,"title":7,"seoTitle":8,"description":9,"summary":10,"primaryKeyword":11,"cluster":12,"published":13,"updated":13,"reviewed":13,"readingMinutes":14,"html":15,"toc":16,"wordCount":32,"sources":33,"faq":49,"related":56,"tested":73,"internalLinks":77},"guides","tool-calling-llm-api","\u002Fguides\u002Ftool-calling-llm-api\u002F","Tool calling with an OpenAI-compatible LLM API","LLM Tool Calling Over an OpenAI-Compatible API","Tool calling with open-weight and uncensored LLMs: defining tools, the request and response loop, validating arguments, limits on rounds, and provider differences.","Describe each function with a JSON Schema in the tools array. When the model wants one, the reply has finish_reason tool_calls and a list of calls with JSON arguments. Run the function yourself, append the result as a tool message with the matching tool_call_id, and call the API again until the model answers in text. Validate arguments and cap the number of rounds.","tool calling llm api","api-features","2026-11-14",2,"\u003Ch2 id=\"how-the-loop-works\">How the loop works\u003C\u002Fh2>\n\u003Cp>Tool calling, also called function calling, lets a model ask your code to do something it cannot do itself, such as look up data or run a calculation. The model never executes anything. A round trip looks like this:\u003C\u002Fp>\n\u003Col>\n\u003Cli>You send messages plus a \u003Ccode>tools\u003C\u002Fcode> array, each tool with a name, a description and a JSON Schema for its parameters.\u003C\u002Fli>\n\u003Cli>The model either answers in text, or returns \u003Ccode>finish_reason: "tool_calls"\u003C\u002Fcode> with one or more calls, each with an \u003Ccode>id\u003C\u002Fcode>, a function name and arguments as a JSON string.\u003C\u002Fli>\n\u003Cli>Your code validates the arguments, runs the function, and appends two things to the conversation: the assistant message with the calls, and a \u003Ccode>tool\u003C\u002Fcode> message per call containing the result and the matching \u003Ccode>tool_call_id\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>You call the API again with the longer conversation. Repeat until the model answers in text.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2 id=\"python\">Python\u003C\u002Fh2>\n\u003Cdiv class=\"code-block\" data-lang=\"python\">\u003Cspan class=\"sr-only\">python code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-python\">"""A complete tool-calling loop: the model asks for a function, we run it,\nsend the result back, and the model writes the final answer."""\nimport json\nimport os\n\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"], timeout=60.0)\nMODEL = os.environ["LLM_MODEL"]\n\ntools = [\n    {\n        "type": "function",\n        "function": {\n            "name": "get_weather",\n            "description": "Current weather for a city.",\n            "parameters": {\n                "type": "object",\n                "properties": {"city": {"type": "string", "description": "City name, e.g. Lisbon"}},\n                "required": ["city"],\n                "additionalProperties": False,\n            },\n        },\n    }\n]\n\n\ndef get_weather(city: str) -> dict:\n    # Replace with a real lookup. Never execute code or shell commands that a model writes.\n    return {"city": city, "temperature_c": 19, "conditions": "clear"}\n\n\nmessages = [{"role": "user", "content": "Should I take an umbrella in Lisbon today?"}]\n\nfor _ in range(5):  # hard limit on round trips\n    reply = client.chat.completions.create(model=MODEL, messages=messages, tools=tools, tool_choice="auto")\n    message = reply.choices[0].message\n    if not message.tool_calls:\n        print(message.content)\n        break\n\n    messages.append(message.model_dump(exclude_none=True))\n    for call in message.tool_calls:\n        if call.function.name != "get_weather":\n            result = {"error": f"unknown tool {call.function.name}"}\n        else:\n            try:\n                args = json.loads(call.function.arguments or "{}")\n                result = get_weather(str(args["city"]))\n            except (json.JSONDecodeError, KeyError) as e:\n                result = {"error": f"bad arguments: {e}"}\n        messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})\nelse:\n    print("Stopped after 5 rounds without a final answer.")\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Ch2 id=\"node-js\">Node.js\u003C\u002Fh2>\n\u003Cdiv class=\"code-block\" data-lang=\"javascript\">\u003Cspan class=\"sr-only\">javascript code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-javascript\">\u002F\u002F A complete tool-calling loop with the OpenAI Node SDK.\nimport OpenAI from 'openai'\n\nconst client = new OpenAI({ baseURL: process.env.LLM_BASE_URL, apiKey: process.env.LLM_API_KEY })\nconst model = process.env.LLM_MODEL\n\nconst tools = [\n  {\n    type: 'function',\n    function: {\n      name: 'get_weather',\n      description: 'Current weather for a city.',\n      parameters: {\n        type: 'object',\n        properties: { city: { type: 'string', description: 'City name, e.g. Lisbon' } },\n        required: ['city'],\n        additionalProperties: false,\n      },\n    },\n  },\n]\n\n\u002F\u002F Replace with a real lookup. Never execute code or shell commands that a model writes.\nconst handlers = {\n  get_weather: ({ city }) => ({ city, temperature_c: 19, conditions: 'clear' }),\n}\n\nconst messages = [{ role: 'user', content: 'Should I take an umbrella in Lisbon today?' }]\n\nlet answered = false\nfor (let round = 0; round < 5 && !answered; round++) {\n  const reply = await client.chat.completions.create({ model, messages, tools, tool_choice: 'auto' })\n  const message = reply.choices[0].message\n  if (!message.tool_calls?.length) {\n    console.log(message.content)\n    answered = true\n    break\n  }\n  messages.push(message)\n  for (const call of message.tool_calls) {\n    let result\n    try {\n      const handler = handlers[call.function.name]\n      if (!handler) throw new Error(`unknown tool ${call.function.name}`)\n      result = handler(JSON.parse(call.function.arguments || '{}'))\n    } catch (err) {\n      result = { error: err.message }\n    }\n    messages.push({ role: 'tool', tool_call_id: call.id, content: JSON.stringify(result) })\n  }\n}\nif (!answered) console.log('Stopped after 5 rounds without a final answer.')\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>Both versions do the same things on purpose:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Cap the rounds.\u003C\u002Fstrong> A model can keep calling tools; five rounds is a reasonable ceiling for most tasks.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Parse arguments defensively.\u003C\u002Fstrong> The arguments are a string the model wrote. It can be invalid JSON or miss a required field. Return an error message as the tool result instead of crashing, so the model can try again.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Allow-list functions.\u003C\u002Fstrong> Only names you defined are run; anything else gets an error result.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Never run model-written code or shell commands\u003C\u002Fstrong> without a sandbox and a human decision. A refusal-reduced model will follow instructions it finds in documents or web pages, which makes prompt injection through tool results a real risk.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"writing-good-tool-definitions\">Writing good tool definitions\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>Give each tool one job and a description that says when to use it.\u003C\u002Fli>\n\u003Cli>Use \u003Ccode>required\u003C\u002Fcode> and \u003Ccode>additionalProperties: false\u003C\u002Fcode> in the schema. Some servers enforce the schema; many open-weight stacks do not, which is why validation in your code still matters.\u003C\u002Fli>\n\u003Cli>Keep the result small and factual. Large tool results use context and cost input tokens on every later round.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"provider-and-model-differences\">Provider and model differences\u003C\u002Fh2>\n\u003Cp>Whether tool calling works depends on three layers: the model’s training, the chat template, and the server’s parser that turns the model’s text into structured \u003Ccode>tool_calls\u003C\u002Fcode>. Model cards often document the parser: GLM-4.7-Flash’s card, for example, serves it with a dedicated GLM tool-call parser in vLLM and SGLang. A provider can accept the \u003Ccode>tools\u003C\u002Fcode> parameter while the model behind it handles tools poorly.\u003C\u002Fp>\n\u003Cp>Documented differences we found:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>MolochAI supports \u003Ccode>tools\u003C\u002Fcode> and \u003Ccode>tool_choice\u003C\u002Fcode>, says quality can vary by server, and sends streamed tool arguments whole rather than in fragments.\u003C\u002Fli>\n\u003Cli>Featherless, Venice and OpenRouter flag tool support per model in their model lists, so check the specific model.\u003C\u002Fli>\n\u003Cli>In the OpenAI streaming format, arguments can arrive in fragments to concatenate by \u003Ccode>index\u003C\u002Fcode>; see the \u003Ca href=\"\u002Fguides\u002Fstreaming-llm-responses\u002F\">streaming guide\u003C\u002Fa>.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Test tool calls with your own tools before committing to a model. Our \u003Ca href=\"\u002Fmodels\u002Fcoding\u002F\">coding model page\u003C\u002Fa> lists which catalogued models’ cards mention tool use, and \u003Ca href=\"\u002Flearn\u002Fevaluate-llm-for-coding\u002F\">how to evaluate an LLM for coding\u003C\u002Fa> covers testing tool calls as part of a task set.\u003C\u002Fp>\n",[17,20,23,26,29],{"id":18,"text":19,"level":14},"how-the-loop-works","How the loop works",{"id":21,"text":22,"level":14},"python","Python",{"id":24,"text":25,"level":14},"node-js","Node.js",{"id":27,"text":28,"level":14},"writing-good-tool-definitions","Writing good tool definitions",{"id":30,"text":31,"level":14},"provider-and-model-differences","Provider and model differences",510,[34,38,41,45],{"title":35,"publisher":36,"url":37},"Function calling guide","OpenAI","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Ffunction-calling",{"title":39,"publisher":36,"url":40},"Chat Completions API reference","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fapi-reference\u002Fchat",{"title":42,"publisher":43,"url":44},"OpenAI compatibility (tool notes)","MolochAI","https:\u002F\u002Fmolochai.dev\u002Fdocs\u002Fopenai-compatibility",{"title":46,"publisher":47,"url":48},"zai-org\u002FGLM-4.7-Flash (serving examples with a tool-call parser)","Hugging Face","https:\u002F\u002Fhuggingface.co\u002Fzai-org\u002FGLM-4.7-Flash",[50,53],{"q":51,"a":52},"Does a refusal-reduced model call tools better?","There is no evidence that it does. Tool calling depends on the base model's training and on the server's parser; reducing refusals does not add either.",{"q":54,"a":55},"Can the model run my functions directly?","No. The model only proposes a call with arguments. Your code decides whether to run it, which is where you enforce limits and permissions.",[57,61,65,69],{"path":58,"title":59,"description":60},"\u002Fmodels\u002Fcoding\u002F","Uncensored AI for coding","Coding criteria and which claims are measured or not.",{"path":62,"title":63,"description":64},"\u002Fguides\u002Fopenai-compatible-api\u002F","OpenAI-compatible LLM APIs, and where they differ","What OpenAI-compatible means for uncensored LLM APIs: base URLs, ignored parameters, reasoning fields, usage in streams, tool calls, plus a script to probe any endpoint.",{"path":66,"title":67,"description":68},"\u002Fguides\u002Fstreaming-llm-responses\u002F","Streaming LLM responses with server-sent events","How LLM API streaming works: the server-sent events format, parsing chunks, the final usage chunk, cancelling a stream, timeouts and errors that arrive mid-stream.",{"path":70,"title":71,"description":72},"\u002Flearn\u002Fevaluate-llm-for-coding\u002F","How to evaluate an LLM for coding","A practical method to evaluate an LLM for coding: real tasks, unit tests, pass@k, contamination checks, cost per solved task, latency, tool calls and run logs.",{"summary":74,"ranOn":13,"environment":75,"liveEndpoint":76},"The Python and Node.js loops were run unchanged against the mock OpenAI-compatible server in this site's repository, which returns a tool call and then a final answer. Tool-calling quality of real models was not tested.","Python 3.12.4 with openai 3.51.0; Node.js 22.19.0 with openai 7.57.0",false,[66,58,70],1794622336928]