How to Build a Text Summarization API With an LLM

The shape

POST /summarize takes {"text": "..."}, calls the LLM with a summarization instruction, and returns {"summary": "..."}. The LLM does the work - your job is a clean endpoint and a good prompt.

The endpoint

import requests
from flask import Flask, request, jsonify

app = Flask(__name__)
LLM_URL = "http://llm-service:8080/v1/chat/completions"

@app.post("/summarize")
def summarize():
    text = (request.get_json() or {}).get("text", "").strip()
    if not text:
        return jsonify({"error": "text is required"}), 400

    payload = {
        "model": "the-model",
        "messages": [
            {"role": "system", "content": "You summarize text concisely and accurately. "
                                          "Return only the summary, no preamble."},
            {"role": "user", "content": f"Summarize the following:\n\n{text}"},
        ],
    }
    r = requests.post(LLM_URL, json=payload, timeout=60)
    r.raise_for_status()
    summary = r.json()["choices"][0]["message"]["content"]
    return jsonify({"summary": summary})

What makes it good

• A system prompt that constrains the output - "return only the summary, no preamble" stops the model from adding "Here's a summary:" noise.
• Validate input - reject empty bodies with 400; that's the most common caller bug.
• Control length - ask for "in 2-3 sentences" or "in under 50 words" if callers need a bounded summary; pass it through as a parameter.
• Timeout + error handling - LLM calls are slow and occasionally fail; set a timeout and return a clean 502/error rather than hanging.
• Long inputs - if text can exceed the model's context window, chunk it and summarize the chunks, then summarize the summaries (map-reduce).

Long inputs and consistent quality

Two things separate a toy summarizer from a usable one. Long inputs: any document longer than the model's context window must be chunked - split on paragraph boundaries, summarize each chunk, then summarize the summaries (a map-reduce pass). Track token counts so an oversized request fails fast with a clear error instead of a truncated, misleading summary.

Consistent output: put the rules in the system prompt, not the user text - length ("3 sentences" or "5 bullets"), audience, and tone. Use a low temperature so the same input gives a stable summary, and return structured output so callers don't parse free text. Guard against prompt injection from the document itself by clearly delimiting the content and instructing the model to treat it as data, not instructions. Add a timeout and a graceful error so one slow or oversized request can't hang the endpoint.