How to Stream Tokens Over Server-Sent Events in FastAPI

Why streaming beats buffering for LLM output

A buffered endpoint waits for the entire response before sending anything. With an LLM generating tokens one at a time over hundreds of milliseconds, that means the user stares at a spinner until the model finishes. SSE flips this: the server pushes each token as it lands, so text appears word by word - the same UX as ChatGPT or Claude.

SSE is the simplest streaming primitive available over plain HTTP. The browser's built-in EventSource reads it without any extra library, and it auto-reconnects on drop. No WebSocket upgrade needed.

The SSE wire format

Each SSE message is a small text block separated by a blank line:

event: token
data: {"text": "hello"}

event: token
data: {"text": "world"}

event: done
data: {"total_tokens": 2}

The event: line is the named event type; data: is the payload (any string, usually JSON). A double newline ends each message - that blank line is mandatory.

Implementing it in FastAPI

FastAPI's StreamingResponse wraps any Python generator and streams its output directly to the client.

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

def generate_tokens(prompt: str):
    # Replace with your real LLM call that yields tokens
    for word in prompt.split():
        yield word

def _sse(event: str, data: dict) -> str:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n"

@app.post("/api/generate")
def generate(body: dict):
    prompt = body.get("prompt", "")

    def stream():
        count = 0
        for tok in generate_tokens(prompt):
            count += 1
            yield _sse("token", {"text": tok})
        yield _sse("done", {"total_tokens": count})

    return StreamingResponse(stream(), media_type="text/event-stream")

Three things make this work:

• media_type="text/event-stream" sets the Content-Type header that tells browsers this is SSE.
• The inner stream() generator yields strings - FastAPI never buffers them, it writes each yield straight to the socket.
• The double \n\n at the end of each _sse() call closes the event block; without it, the client never fires the event.

Testing it on the command line

# -N disables curl's own output buffering so you see events as they arrive
curl -N -X POST -H 'Content-Type: application/json' \
  -d '{"prompt": "hello world from sse"}' \
  http://localhost:8000/api/generate

You should see event: token lines appear one at a time, then a final event: done.

Reading SSE in the browser

// EventSource only handles GET; for POST you need fetch with a reader
const res = await fetch("/api/generate", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt: "hello" }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = "";
while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buf += decoder.decode(value);
    // parse complete SSE messages from buf as needed
}

For GET-based streams, new EventSource("/stream") handles reconnection and named events automatically.