[{"data":1,"prerenderedAt":83},["ShallowReactive",2],{"guides-streaming-llm-responses":3},{"collection":4,"slug":5,"path":6,"title":7,"seoTitle":8,"description":9,"summary":10,"primaryKeyword":11,"cluster":12,"published":13,"updated":13,"reviewed":13,"readingMinutes":14,"html":15,"toc":16,"wordCount":36,"sources":37,"faq":50,"related":60,"tested":77,"internalLinks":81},"guides","streaming-llm-responses","\u002Fguides\u002Fstreaming-llm-responses\u002F","Streaming LLM responses with server-sent events","LLM API Streaming Tutorial: SSE, Cancel and Timeouts","How LLM API streaming works: the server-sent events format, parsing chunks, the final usage chunk, cancelling a stream, timeouts and errors that arrive mid-stream.","With stream set to true, an OpenAI-compatible API sends the reply as server-sent events: lines that start with \"data:\", each holding a JSON chunk with a small delta of text, ending with \"data: [DONE]\". Read the body incrementally, buffer until a blank line, parse each event, and cancel the request when you no longer need the rest.","llm api streaming tutorial","api-features","2026-11-14",3,"\u003Ch2 id=\"why-stream\">Why stream\u003C\u002Fh2>\n\u003Cp>A non-streamed request returns nothing until the whole reply is ready. For a long answer, or a thinking model that reasons for many seconds first, that is a long blank wait. Streaming sends text as it is generated, lets you show progress, and lets you stop early when the answer is clearly wrong or the user cancels.\u003C\u002Fp>\n\u003Ch2 id=\"what-arrives-on-the-wire\">What arrives on the wire\u003C\u002Fh2>\n\u003Cp>Set \u003Ccode>"stream": true\u003C\u002Fcode> and the response has the content type \u003Ccode>text\u002Fevent-stream\u003C\u002Fcode>. The body follows the server-sent events format from the HTML standard: each event is one or more lines, and a blank line ends an event. Chat-completion servers use a single field, \u003Ccode>data:\u003C\u002Fcode>, holding JSON:\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"text\">\u003Cspan class=\"sr-only\">text code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-text\">data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}\n\ndata: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Mock "},"finish_reason":null}]}\n\ndata: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}\n\ndata: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":6,"completion_tokens":10,"total_tokens":16}}\n\ndata: [DONE]\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>These lines are from our mock server, shortened. The lifecycle is the same on real providers:\u003C\u002Fp>\n\u003Col>\n\u003Cli>A first chunk with the role and usually empty content.\u003C\u002Fli>\n\u003Cli>Many chunks, each with a small \u003Ccode>delta.content\u003C\u002Fcode>. Join them to get the reply.\u003C\u002Fli>\n\u003Cli>A chunk with \u003Ccode>finish_reason\u003C\u002Fcode> (\u003Ccode>stop\u003C\u002Fcode>, \u003Ccode>length\u003C\u002Fcode> or \u003Ccode>tool_calls\u003C\u002Fcode>) and an empty delta.\u003C\u002Fli>\n\u003Cli>Optionally, a usage chunk with an empty \u003Ccode>choices\u003C\u002Fcode> array. OpenAI-style servers send it when you set \u003Ccode>stream_options: {"include_usage": true}\u003C\u002Fcode>; some providers always send it, and some never do.\u003C\u002Fli>\n\u003Cli>The literal \u003Ccode>data: [DONE]\u003C\u002Fcode>, after which the server closes the connection.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2 id=\"parsing-a-stream-by-hand\">Parsing a stream by hand\u003C\u002Fh2>\n\u003Cp>The SDKs do this for you, but writing it once shows what can go wrong. This Node.js example needs no dependencies:\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"javascript\">\u003Cspan class=\"sr-only\">javascript code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-javascript\">\u002F\u002F Streaming without an SDK: fetch, parse server-sent events by hand, cancel\n\u002F\u002F early. Runs on Node 18+ and in browsers (but never put an API key in a browser).\nconst res = await fetch(`${process.env.LLM_BASE_URL}\u002Fchat\u002Fcompletions`, {\n  method: 'POST',\n  headers: { Authorization: `Bearer ${process.env.LLM_API_KEY}`, 'Content-Type': 'application\u002Fjson' },\n  body: JSON.stringify({\n    model: process.env.LLM_MODEL,\n    messages: [{ role: 'user', content: 'List three uses of a hash map.' }],\n    max_tokens: 200,\n    stream: true,\n  }),\n  signal: AbortSignal.timeout(30_000),\n})\nif (!res.ok) {\n  console.error(`HTTP ${res.status}: ${await res.text()}`)\n  process.exit(1)\n}\n\nconst decoder = new TextDecoder()\nlet buffer = ''\nlet text = ''\nlet done = false\nconst reader = res.body.getReader()\n\nwhile (!done) {\n  const { value, done: ended } = await reader.read()\n  if (ended) break\n  buffer += decoder.decode(value, { stream: true })\n  \u002F\u002F Events are separated by a blank line; a chunk can end mid-event.\n  let boundary\n  while ((boundary = buffer.indexOf('\\n\\n')) !== -1) {\n    const event = buffer.slice(0, boundary)\n    buffer = buffer.slice(boundary + 2)\n    for (const line of event.split('\\n')) {\n      if (!line.startsWith('data:')) continue \u002F\u002F ignore comments and other fields\n      const data = line.slice(5).trim()\n      if (data === '[DONE]') {\n        done = true\n        break\n      }\n      const chunk = JSON.parse(data)\n      if (chunk.error) throw new Error(chunk.error.message) \u002F\u002F some servers send errors mid-stream\n      const piece = chunk.choices?.[0]?.delta?.content\n      if (piece) {\n        text += piece\n        process.stdout.write(piece)\n      }\n    }\n  }\n}\nawait reader.cancel() \u002F\u002F close the connection if we stopped before the server did\nprocess.stdout.write('\\n')\nconsole.log(`characters received: ${text.length}`)\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>The important details:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Buffer across reads.\u003C\u002Fstrong> Network chunks do not line up with events. A read can end halfway through a JSON line, so keep the remainder and only parse complete events (text up to a blank line).\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Ignore what you do not know.\u003C\u002Fstrong> Lines that do not start with \u003Ccode>data:\u003C\u002Fcode> can be comments (\u003Ccode>: keep-alive\u003C\u002Fcode>) or other fields. Skipping them is correct per the standard.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Stop on \u003Ccode>[DONE]\u003C\u002Fcode>.\u003C\u002Fstrong> It is not JSON, so test for it before parsing.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Errors can arrive inside the stream.\u003C\u002Fstrong> Once the server has sent a 200 status, it cannot change it. Some servers report a later failure as an event carrying an \u003Ccode>error\u003C\u002Fcode> object, which is why the parser checks for one. MolochAI documents this behaviour for failures after the stream opens.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"cancelling-and-timeouts\">Cancelling and timeouts\u003C\u002Fh2>\n\u003Cp>Cancel by aborting the request. With \u003Ccode>fetch\u003C\u002Fcode>, pass an \u003Ccode>AbortSignal\u003C\u002Fcode>; \u003Ccode>AbortSignal.timeout(30_000)\u003C\u002Fcode> gives up after 30 seconds. With the Node SDK, pass \u003Ccode>{ signal }\u003C\u002Fcode> as the second argument; the \u003Ca href=\"\u002Fguides\u002Fnodejs-uncensored-llm-api\u002F\">Node.js guide\u003C\u002Fa> shows a timer that aborts a slow stream. In Python, call \u003Ccode>stream.close()\u003C\u002Fcode> or leave a \u003Ccode>with\u003C\u002Fcode> block; the \u003Ca href=\"\u002Fguides\u002Fpython-uncensored-llm-api\u002F\">Python guide\u003C\u002Fa> uses \u003Ccode>try\u003C\u002Fcode>\u002F\u003Ccode>finally\u003C\u002Fcode> so the connection is released even if your loop breaks early.\u003C\u002Fp>\n\u003Cp>Set two kinds of limit:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>A total timeout\u003C\u002Fstrong> for the whole request, so a stuck connection does not hang your app.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>An idle timeout\u003C\u002Fstrong> between chunks if your runtime supports it. A thinking model can be silent for a while before the first visible token, so make this generous for reasoning models.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"tool-calls-in-a-stream\">Tool calls in a stream\u003C\u002Fh2>\n\u003Cp>When a model calls a tool while streaming, the chunks carry \u003Ccode>delta.tool_calls\u003C\u002Fcode> instead of content. In the OpenAI format the function arguments can arrive in fragments that you concatenate by \u003Ccode>index\u003C\u002Fcode> until \u003Ccode>finish_reason\u003C\u002Fcode> is \u003Ccode>tool_calls\u003C\u002Fcode>; some providers send the arguments in one piece instead. Code that concatenates fragments handles both. The \u003Ca href=\"\u002Fguides\u002Ftool-calling-llm-api\u002F\">tool-calling guide\u003C\u002Fa> covers the full loop.\u003C\u002Fp>\n\u003Ch2 id=\"checklist\">Checklist\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>Skip chunks whose \u003Ccode>choices\u003C\u002Fcode> array is empty, except to read \u003Ccode>usage\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>Treat \u003Ccode>finish_reason: "length"\u003C\u002Fcode> as a cut-off reply.\u003C\u002Fli>\n\u003Cli>Always close or cancel the stream when you stop reading.\u003C\u002Fli>\n\u003Cli>Never stream from the browser with your own API key: proxy through your server.\u003C\u002Fli>\n\u003C\u002Ful>\n",[17,21,24,27,30,33],{"id":18,"text":19,"level":20},"why-stream","Why stream",2,{"id":22,"text":23,"level":20},"what-arrives-on-the-wire","What arrives on the wire",{"id":25,"text":26,"level":20},"parsing-a-stream-by-hand","Parsing a stream by hand",{"id":28,"text":29,"level":20},"cancelling-and-timeouts","Cancelling and timeouts",{"id":31,"text":32,"level":20},"tool-calls-in-a-stream","Tool calls in a stream",{"id":34,"text":35,"level":20},"checklist","Checklist",597,[38,42,46],{"title":39,"publisher":40,"url":41},"Server-sent events","WHATWG HTML Living Standard","https:\u002F\u002Fhtml.spec.whatwg.org\u002Fmultipage\u002Fserver-sent-events.html",{"title":43,"publisher":44,"url":45},"Chat Completions API reference (streaming)","OpenAI","https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fapi-reference\u002Fchat",{"title":47,"publisher":48,"url":49},"OpenAI compatibility (streaming notes)","MolochAI","https:\u002F\u002Fmolochai.dev\u002Fdocs\u002Fopenai-compatibility",[51,54,57],{"q":52,"a":53},"Does streaming cost more?","No. You are billed for the same input and output tokens. Cancelling a stream early can save output tokens, depending on how quickly the provider stops generating.",{"q":55,"a":56},"Why is the first chunk empty?","Many servers send a first chunk with only the role, and an empty content string, before any text. Skip deltas without content.",{"q":58,"a":59},"Can I use the browser EventSource API?","EventSource only sends GET requests and cannot set an Authorization header, so it does not fit chat completions. Use fetch with a streamed body, from your own server.",[61,65,69,73],{"path":62,"title":63,"description":64},"\u002Fguides\u002Fnodejs-uncensored-llm-api\u002F","Call an uncensored LLM API from Node.js","Call an uncensored or open-weight LLM from Node.js with the OpenAI SDK: environment variables, a first request, streaming with cancellation, errors and retries.",{"path":66,"title":67,"description":68},"\u002Fguides\u002Fpython-uncensored-llm-api\u002F","Call an uncensored LLM API from Python","Call an uncensored or open-weight LLM from Python with the OpenAI SDK: environment variables, first request, streaming, error handling, retries and provider switching.",{"path":70,"title":71,"description":72},"\u002Fguides\u002Fopenai-compatible-api\u002F","OpenAI-compatible LLM APIs, and where they differ","What OpenAI-compatible means for uncensored LLM APIs: base URLs, ignored parameters, reasoning fields, usage in streams, tool calls, plus a script to probe any endpoint.",{"path":74,"title":75,"description":76},"\u002Ftools\u002Fcontext-window-planner\u002F","Context window planner","Budget input, output and headroom for one request.",{"summary":78,"ranOn":13,"environment":79,"liveEndpoint":80},"The fetch-based parser and the SDK examples were run unchanged against the mock OpenAI-compatible server in this site's repository, which streams real SSE events with a final usage chunk.","Node.js 22.19.0 (fetch parser and openai 7.57.0), Python 3.12.4 (openai 3.51.0)",false,[62,66,82],"\u002Fguides\u002Ftool-calling-llm-api\u002F",1794622336928]