How to Fix a Broken LLM API Integration

The two failures, and what they mean

• Connection error / timeout -> you're hitting the wrong host, port, or path. The request never reaches a working endpoint.
• "model not found" / 400 / 404 from the API -> the request arrived but the body is wrong - usually a missing or wrong model field, or the wrong route (/v1/completions vs /v1/chat/completions).

Read the exact error first - it tells you which of the two you have.

The correct request

Modern chat models use /v1/chat/completions with a model and a messages array:

import requests

LLM_URL = "http://llm-service:8080/v1/chat/completions"   # right host:port + path

def summarize(text: str) -> str:
    payload = {
        "model": "the-model-name",                 # REQUIRED - omitting it = "model not found"
        "messages": [{"role": "user", "content": f"Summarize:\n{text}"}],
    }
    r = requests.post(LLM_URL, json=payload, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

Debug it methodically

1. Curl the endpoint to separate network from payload: bash curl -sS http://llm-service:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"the-model-name","messages":[{"role":"user","content":"hi"}]}' - Connection refused -> wrong host/port. 404 -> wrong path. 400/"model" -> body issue.
2. Check the path - chat models are /v1/chat/completions; the older text route is /v1/completions (different body shape).
3. Check required fields - model is required; messages (chat) vs prompt (completions) must match the route.
4. Parse the right field - the reply is at choices[0].message.content for chat.

Make it robust

• Always set a timeout - LLMs are slow and a hang shouldn't freeze your app.
• raise_for_status() + retry on 429/5xx with backoff.
• Guard the response parse so a malformed reply doesn't crash the caller.