How to Handle LLM API Errors (Retries + Timeouts)

Why one error kills the whole run

A naive pipeline loops over items, calls the LLM, and appends results. With no error handling, the first timeout or 503 throws, the loop dies, and everything processed so far is lost. LLM APIs are slow and occasionally flaky, so this will happen - resilience isn't optional for batch work.

1. A real timeout

The default (or a too-short timeout) means a slow response hangs or fails the call. Set one that matches how long generation actually takes:

resp = requests.post(LLM_URL, json=payload, timeout=60)

2. try/except per item + 3. retry transient errors

Wrap each call so a failure is caught, and retry transient failures (timeouts, 429, 5xx) with a short backoff before giving up on that item:

def call_with_retry(payload, attempts=3):
    for i in range(attempts):
        try:
            r = requests.post(LLM_URL, json=payload, timeout=60)
            r.raise_for_status()
            return r.json()
        except (requests.Timeout, requests.HTTPError, requests.ConnectionError) as e:
            if i == attempts - 1:
                raise
            time.sleep(2 ** i)        # 1s, 2s, 4s backoff

4. Partial results - never lose the whole run

Process each item independently; on permanent failure, record the error and keep going so the rest of the batch still completes:

results = []
for item in items:
    try:
        results.append({"id": item["id"], "tags": call_with_retry(build(item))})
    except Exception as e:
        results.append({"id": item["id"], "error": str(e)})   # don't abort the batch

The principles

• Retry only transient errors (timeout, 429, 5xx) - a 400 (bad request) won't fix itself, so don't waste retries on it.
• Exponential backoff spaces retries out so you don't hammer a struggling API.
• Idempotent + resumable: save progress as you go so a re-run skips finished items.
• Isolate failures: one bad item shouldn't take down the other 999.