How to Call an LLM from a Chatbot (AWS Bedrock)

The request shape

Most LLM chat APIs (AWS Bedrock's OpenAI-compatible endpoint, the OpenAI API, and local servers) take the same messages array and return the same choices shape:

import requests

LLM_URL = "https://<bedrock-endpoint>/v1/chat/completions"
MODEL = "anthropic.claude-3-haiku"

def chat(user_msg: str) -> str:
    r = requests.post(LLM_URL, json={
        "model": MODEL,
        "messages": [{"role": "user", "content": user_msg}],
    }, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

The reply text is at choices[0].message.content - that's the one path to remember.

Add a system prompt + history for a real chatbot

A single user message is stateless. A chatbot sends a system message (its persona/ rules) plus the running conversation history:

messages = [
    {"role": "system", "content": "You are a helpful support agent. Be concise."},
    *history,                                   # prior user/assistant turns
    {"role": "user", "content": user_msg},
]

Append each user message and the model's reply back into history so the next turn has context.

Error handling that matters

• Timeout - LLM calls are slow; always set a timeout and handle it (retry or a graceful "try again" message). A hung request shouldn't hang your app.
• Non-200 - raise_for_status() surfaces rate limits (429) and server errors (5xx); back off and retry on those.
• Bad/empty response - guard the choices[0]... access so a malformed reply doesn't crash the handler.

Streaming (optional)

For a typing-effect UI, request "stream": true and read the response incrementally (server-sent events) instead of waiting for the whole reply - the API returns token chunks you forward to the client as they arrive.