Streaming LLM responses with server-sent events

Why stream

A non-streamed request returns nothing until the whole reply is ready. For a long answer, or a thinking model that reasons for many seconds first, that is a long blank wait. Streaming sends text as it is generated, lets you show progress, and lets you stop early when the answer is clearly wrong or the user cancels.

What arrives on the wire

Set "stream": true and the response has the content type text/event-stream. The body follows the server-sent events format from the HTML standard: each event is one or more lines, and a blank line ends an event. Chat-completion servers use a single field, data:, holding JSON:

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Mock "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":6,"completion_tokens":10,"total_tokens":16}}

data: [DONE]

These lines are from our mock server, shortened. The lifecycle is the same on real providers:

1. A first chunk with the role and usually empty content.
2. Many chunks, each with a small delta.content. Join them to get the reply.
3. A chunk with finish_reason (stop, length or tool_calls) and an empty delta.
4. Optionally, a usage chunk with an empty choices array. OpenAI-style servers send it when you set stream_options: {"include_usage": true}; some providers always send it, and some never do.
5. The literal data: [DONE], after which the server closes the connection.

Parsing a stream by hand

The SDKs do this for you, but writing it once shows what can go wrong. This Node.js example needs no dependencies:

// Streaming without an SDK: fetch, parse server-sent events by hand, cancel
// early. Runs on Node 18+ and in browsers (but never put an API key in a browser).
const res = await fetch(`${process.env.LLM_BASE_URL}/chat/completions`, {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.LLM_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: process.env.LLM_MODEL,
    messages: [{ role: 'user', content: 'List three uses of a hash map.' }],
    max_tokens: 200,
    stream: true,
  }),
  signal: AbortSignal.timeout(30_000),
})
if (!res.ok) {
  console.error(`HTTP ${res.status}: ${await res.text()}`)
  process.exit(1)
}

const decoder = new TextDecoder()
let buffer = ''
let text = ''
let done = false
const reader = res.body.getReader()

while (!done) {
  const { value, done: ended } = await reader.read()
  if (ended) break
  buffer += decoder.decode(value, { stream: true })
  // Events are separated by a blank line; a chunk can end mid-event.
  let boundary
  while ((boundary = buffer.indexOf('\n\n')) !== -1) {
    const event = buffer.slice(0, boundary)
    buffer = buffer.slice(boundary + 2)
    for (const line of event.split('\n')) {
      if (!line.startsWith('data:')) continue // ignore comments and other fields
      const data = line.slice(5).trim()
      if (data === '[DONE]') {
        done = true
        break
      }
      const chunk = JSON.parse(data)
      if (chunk.error) throw new Error(chunk.error.message) // some servers send errors mid-stream
      const piece = chunk.choices?.[0]?.delta?.content
      if (piece) {
        text += piece
        process.stdout.write(piece)
      }
    }
  }
}
await reader.cancel() // close the connection if we stopped before the server did
process.stdout.write('\n')
console.log(`characters received: ${text.length}`)

The important details:

• Buffer across reads. Network chunks do not line up with events. A read can end halfway through a JSON line, so keep the remainder and only parse complete events (text up to a blank line).
• Ignore what you do not know. Lines that do not start with data: can be comments (: keep-alive) or other fields. Skipping them is correct per the standard.
• Stop on [DONE]. It is not JSON, so test for it before parsing.
• Errors can arrive inside the stream. Once the server has sent a 200 status, it cannot change it. Some servers report a later failure as an event carrying an error object, which is why the parser checks for one. MolochAI documents this behaviour for failures after the stream opens.

Cancelling and timeouts

Cancel by aborting the request. With fetch, pass an AbortSignal; AbortSignal.timeout(30_000) gives up after 30 seconds. With the Node SDK, pass { signal } as the second argument; the Node.js guide shows a timer that aborts a slow stream. In Python, call stream.close() or leave a with block; the Python guide uses try/finally so the connection is released even if your loop breaks early.

Set two kinds of limit:

• A total timeout for the whole request, so a stuck connection does not hang your app.
• An idle timeout between chunks if your runtime supports it. A thinking model can be silent for a while before the first visible token, so make this generous for reasoning models.

Tool calls in a stream

When a model calls a tool while streaming, the chunks carry delta.tool_calls instead of content. In the OpenAI format the function arguments can arrive in fragments that you concatenate by index until finish_reason is tool_calls; some providers send the arguments in one piece instead. Code that concatenates fragments handles both. The tool-calling guide covers the full loop.

Checklist

• Skip chunks whose choices array is empty, except to read usage.
• Treat finish_reason: "length" as a cut-off reply.
• Always close or cancel the stream when you stop reading.
• Never stream from the browser with your own API key: proxy through your server.