How to Stream LLM Tokens With Server-Sent Events

Q: What format does Server-Sent Events use?

Each event is `data: \n\n` sent with the `text/event-stream` content type. The browser's built-in EventSource consumes it and fires a message event per chunk.

Streaming makes a chat reply appear word by word instead of after a long pause. The catch: you must yield each chunk as it arrives, not collect them all then send once. Here's how to stream LLM tokens over SSE.

AI Engineerpythonflaskllm

Why the response appears all at once

The classic bug: the endpoint reads the LLM's streamed chunks into a string and yields it once at the end. That defeats streaming entirely - the user waits for the whole generation, same as non-streaming. To stream, you must forward each chunk the moment it arrives.

Request a streaming completion and forward each chunk

Set stream: true, iterate the response line by line, and yield each token as an SSE event:

import json, requests
from flask import Response

@app.post("/chat/stream")
def chat_stream():
    def generate():
        payload = {"model": "the-model", "messages": [...], "stream": True}
        with requests.post(LLM_URL, json=payload, stream=True, timeout=60) as r:
            for line in r.iter_lines():                 # arrives incrementally
                if not line:
                    continue
                decoded = line.decode("utf-8")
                if not decoded.startswith("data: "):
                    continue
                data = decoded[6:]
                if data.strip() == "[DONE]":
                    break
                delta = json.loads(data)["choices"][0]["delta"].get("content", "")
                if delta:
                    yield f"data: {delta}\n\n"          # forward immediately
    return Response(generate(), mimetype="text/event-stream")

The key is the yield is inside the loop - each token goes out as it's produced, not after the loop finishes.

The details that matter

stream=True on the request so requests doesn't buffer the whole body before you can read it.
SSE format is data: <payload>\n\n and mimetype="text/event-stream"; the browser's EventSource reads it natively.
Disable proxy buffering - nginx will buffer SSE by default and re-introduce the "all at once" delay; set X-Accel-Buffering: no (or proxy_buffering off).
Handle disconnects - if the client closes, stop pulling from the LLM so you don't burn tokens on an abandoned stream.

Want to try it hands-on? HeyDevJob gives you this exact setup in a live cloud workspace in your browser - edit it, run it, and see it work. Free, nothing to install.

Try it in a workspace →

What you'll practice

Yielding each chunk as it arrives (not buffering)
Emitting correct SSE data: frames
Disabling proxy buffering for streaming

FAQ

How do I stream an LLM response token by token?

Request the completion with stream=true, iterate the response as it arrives (stream=True so it isn't buffered), and yield each content delta immediately as an SSE 'data:' event - don't collect everything and send once.

Why does my streamed LLM response still appear all at once?

Either you're buffering the chunks into a string and yielding at the end, or a proxy (like nginx) is buffering the SSE stream. Yield inside the loop, and disable proxy buffering with X-Accel-Buffering: no.

What format does Server-Sent Events use?

Each event is data: <payload>\n\n sent with the text/event-stream content type. The browser's built-in EventSource consumes it and fires a message event per chunk.

Keep learning

Restore a Broken LLM API IntegrationAI/ML project Harden an LLM Pipeline Against API FailuresAI/ML project Parse JSON From an LLM (Strip Markdown Fences)AI/ML project AI/ML roadmapStep by step to hired AI/ML interview questionsSTAR answers All AI/ML projectsProjects hub

Learn it by doing. Open this in a live cloud workspace, make the change yourself, and keep a record of the work you can share.

Open the workspace →