How to Validate an LLM API Response in Python

The symptom: every answer is "failed"

A Flask Q&A bot calls an OpenAI-compatible chat completions endpoint. The model replies with a correct answer, but the bot returns this for every question:

{"status": "failed", "error": "LLM request failed: missing usage data"}

The HTTP call worked. The problem is the check that runs after it:

data = response.json()

if "usage" not in data or "total_tokens" not in data.get("usage", {}):
    raise ValueError("LLM request failed: missing usage data")

answer = data["choices"][0]["message"]["content"].strip()

The code treats usage as proof that the request succeeded. But many OpenAI-compatible servers (local model servers, proxies, gateways, and streaming responses) leave usage out or send it in a different shape. A valid answer is thrown away because of a field the bot never uses.

Look at the raw response first

Before you change validation code, look at what the API really returns. Call the endpoint directly and print the JSON:

curl -s -X POST localhost:5002/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' \
  | python3 -m json.tool

A typical reply looks like this:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "The capital of France is Paris."},
      "finish_reason": "stop"
    }
  ]
}

There is an answer in choices[0].message.content, and there is no usage block. The validation rule is wrong, not the model.

Validate what you use, not what you expect

The rule is simple: a response is valid if it contains the data your code reads next. For a chat bot, that is the first choice and its message content:

def ask_llm(question):
    payload = {
        "model": MODEL,
        "messages": [
            {"role": "system", "content": "Answer in 1-3 sentences."},
            {"role": "user", "content": question},
        ],
        "temperature": 0.5,
    }
    response = requests.post(LLM_URL, json=payload, timeout=60)
    response.raise_for_status()
    data = response.json()

    if not data.get("choices"):
        raise ValueError("LLM request failed: no choices in response")

    content = (data["choices"][0].get("message") or {}).get("content")
    if not content or not content.strip():
        raise ValueError("LLM request failed: empty answer")

    return content.strip()

This version still fails when it should. An empty choices list or an empty answer is a real failure, and the endpoint can return a non-200 status with "status": "failed". A missing usage field is not a failure, so it no longer stops the bot.

Treat optional fields as optional

If you want token counts for logging or cost tracking, read them without making them required:

usage = data.get("usage") or {}
total_tokens = usage.get("total_tokens")
if total_tokens is not None:
    app.logger.info("tokens used: %s", total_tokens)

The same rule applies to other fields that differ between providers, such as id, created, system_fingerprint, or finish_reason. Log them when they exist. Do not reject a response because they are missing.

Keep the error path honest

Good validation separates three different cases, so the caller gets a clear answer:

• Bad input from the user - for example a request body with no question. Return 400 before you call the model.
• The model call failed - a timeout, a connection error, or a non-2xx status from the API. Let raise_for_status() raise, then return a 502 or 500 with an error message.
• The call worked but the answer is unusable - no choices, or empty content. Return a failure status so the client does not show a blank answer.

@app.route("/ask", methods=["POST"])
def ask():
    body = request.get_json(silent=True) or {}
    question = body.get("question")
    if not question:
        return jsonify({"status": "failed", "error": "missing 'question'"}), 400
    try:
        answer = ask_llm(question)
    except (requests.RequestException, ValueError) as e:
        return jsonify({"status": "failed", "error": str(e)}), 502
    return jsonify({"status": "success", "answer": answer})

Going further: validate with a schema

For larger services, describe the response shape once with Pydantic and let it do the checks. Mark optional fields as optional, so the schema matches the real API:

from pydantic import BaseModel

class Message(BaseModel):
    content: str

class Choice(BaseModel):
    message: Message

class ChatResponse(BaseModel):
    choices: list[Choice]
    usage: dict | None = None

parsed = ChatResponse.model_validate(data)
if not parsed.choices:
    raise ValueError("no choices in response")

A schema makes the contract clear to the next developer: which fields the code needs, and which fields it can live without.