How to Route LLM Requests to Cheaper Models

Q: How do I route LLM requests to cheaper models?

Write a pick_model function that inspects the prompt - start with length - and returns the cheap model for short/simple inputs and the expensive model for longer or more complex ones. Call your LLM API with whichever model pick_model returns.

Q: What's a good prompt length threshold for LLM routing?

80-100 characters is a common starting point for separating one-word/short acknowledgements from substantive prompts. Tune it by logging which model was chosen and checking output quality at the boundary.

Q: How do I route by quality, not just length?

Use a confidence fallback: call the cheap model first, and if its response looks low-confidence (e.g. short, uncertain phrasing, or a classifier score below a threshold), retry with the expensive model. Tools like Martian and OpenRouter do this automatically.

Not every LLM request needs the expensive model. A simple router that checks prompt length - or complexity - can send 60-80% of traffic to a cheap model and reserve the powerful one for prompts that actually need it.

AI Engineerllmroutingcost-optimization

Why model routing matters

In any LLM-powered product, a large share of requests are short acknowledgements, yes/no questions, or simple lookups. Sending all of them to the same premium model wastes money and quota. Model routing solves this by inspecting each prompt before dispatch and picking the cheapest model that can handle it.

Even a crude heuristic - route on prompt length - can cut 60-80% of requests off the expensive model without degrading answer quality on simple inputs.

The pattern: pick_model + ask

The core structure is two functions: one that selects the model, one that calls it. Separating them makes the routing logic testable and replaceable without touching the HTTP call.

import requests

LLM_URL        = "http://llm-service:8080/v1/chat/completions"
CHEAP_MODEL    = "fast-model"
EXPENSIVE_MODEL = "capable-model"

last_model_used: str = ""


def pick_model(prompt: str) -> str:
    """Return the cheap model for short prompts, the expensive one otherwise."""
    return CHEAP_MODEL if len(prompt) < 80 else EXPENSIVE_MODEL


def ask(prompt: str) -> str:
    global last_model_used
    model = pick_model(prompt)
    last_model_used = model          # expose the decision for logging/inspection
    r = requests.post(LLM_URL, json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
    }, timeout=60)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

last_model_used is a module-level variable set on every call. Monitoring it lets you see the actual routing distribution in production - a key signal for tuning the threshold.

Choosing a routing heuristic

The simplest heuristics by complexity and cost:

Signal	How to use it	Good for
Prompt length	`len(prompt) < N`	A fast, zero-cost baseline
Token count	tokenize and count	More accurate than chars; small overhead
Category keyword	regex or small classifier	Intent-based routing (code vs. chat)
Confidence fallback	try cheap, retry expensive if confidence is low	Best quality/cost ratio
Daily spend cap	route to cheap when budget is near limit	Hard cost ceilings

Logging the decision

Always record which model was chosen alongside the latency and outcome. Without this you can't tune the threshold or catch regressions when a new model version ships.

import time

def ask_with_log(prompt: str) -> str:
    t0 = time.monotonic()
    reply = ask(prompt)
    print(f"model={last_model_used} chars={len(prompt)} latency={time.monotonic()-t0:.2f}s")
    return reply

Production routers

Tools like Martian, OpenRouter, and Cloudflare AI Gateway add quality-based routing on top of the length heuristic - they send the prompt to the cheap model and, if the response confidence is below a threshold, retry with the expensive one. That pattern gives near-premium quality at near-cheap cost.

Want to try it hands-on? HeyDevJob gives you this exact setup in a live cloud workspace in your browser - edit it, run it, and see it work. Free, nothing to install.

Try it in a workspace →

What you'll practice

Implementing pick_model to route on prompt length
Calling the LLM API with the model pick_model selected
Exposing last_model_used for routing visibility and logging

FAQ

How do I route LLM requests to cheaper models?

Write a pick_model function that inspects the prompt - start with length - and returns the cheap model for short/simple inputs and the expensive model for longer or more complex ones. Call your LLM API with whichever model pick_model returns.

What's a good prompt length threshold for LLM routing?

80-100 characters is a common starting point for separating one-word/short acknowledgements from substantive prompts. Tune it by logging which model was chosen and checking output quality at the boundary.

How do I route by quality, not just length?

Use a confidence fallback: call the cheap model first, and if its response looks low-confidence (e.g. short, uncertain phrasing, or a classifier score below a threshold), retry with the expensive model. Tools like Martian and OpenRouter do this automatically.

Keep learning

Restore a Broken LLM API IntegrationAI/ML project Harden an LLM Pipeline Against API FailuresAI/ML project Parse JSON From an LLM (Strip Markdown Fences)AI/ML project AI/ML roadmapStep by step to hired AI/ML interview questionsSTAR answers All AI/ML projectsProjects hub

Learn it by doing. Open this in a live cloud workspace, make the change yourself, and keep a record of the work you can share.

Open the workspace →