How to Stream an AI Chat Response in Flask

Why it buffers

When you call an LLM API without stream=True, the request blocks until the model finishes generating, then returns the entire response in one payload. Your Flask endpoint reads that single payload and sends it as one HTTP response body - which is why the browser freezes, then gets the full text all at once.

Even if you enable streaming on the LLM side, forgetting to forward each chunk immediately (for example, by accumulating tokens into a string first) has the same effect - the buffering just moves from the LLM layer into your own code.

The fix: stream=True + yield per token

Use requests with stream=True, iterate r.iter_lines(), and yield each SSE chunk the moment it arrives:

import json
import requests
from flask import Flask, request, Response, stream_with_context

app = Flask(__name__)
LLM_URL = "http://llm-service:8080/v1/chat/completions"

@app.post("/api/chat/stream")
def chat_stream():
    message = (request.get_json(silent=True) or {}).get("message", "")

    @stream_with_context
    def gen():
        body = {
            "model": "your-model",
            "messages": [{"role": "user", "content": message}],
            "stream": True,
        }
        with requests.post(LLM_URL, json=body, stream=True, timeout=60) as r:
            for line in r.iter_lines():
                if not line:
                    continue
                decoded = line.decode("utf-8")
                if not decoded.startswith("data: "):
                    continue
                payload = decoded[6:]
                if payload.strip() == "[DONE]":
                    continue
                try:
                    token = json.loads(payload)["choices"][0]["delta"].get("content", "")
                except Exception:
                    continue
                if token:
                    yield f"data: {json.dumps({'content': token})}\n\n"
        yield "data: [DONE]\n\n"

    return Response(gen(), mimetype="text/event-stream")

The key points: - stream=True on the requests.post call keeps the connection open and lets iter_lines() yield each SSE line as it arrives. - @stream_with_context keeps the Flask request context alive for the duration of the generator so request.* access inside gen() is safe. - Yield immediately - do not accumulate tokens into a buffer and flush at the end. - Return mimetype="text/event-stream" so browsers and React's EventSource (or a fetch + ReadableStream reader) interpret each data: line as a chunk.

Reading the stream on the React side

const res = await fetch('/api/chat/stream', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ message }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  for (const line of buffer.split('\n')) {
    if (line.startsWith('data: ') && line !== 'data: [DONE]') {
      const { content } = JSON.parse(line.slice(6));
      appendToChat(content);
    }
  }
  buffer = '';
}

Watch for buffering at every hop

• nginx - set proxy_buffering off; on the location that proxies to Flask, or nginx will collect the stream before it reaches the browser.
• gzip/compression middleware - compressing a stream forces buffering; disable compression for text/event-stream responses.
• CDN / reverse proxy - many CDNs buffer streaming responses by default; check your provider's SSE or streaming docs.