What Is an LLM Gateway? A Plain Explanation

The short version

If you searched "what is llm gateway", here is the answer in one paragraph. A gateway is a service that accepts requests in a common format, usually the OpenAI chat format, and forwards them to a model API. Along the way it checks who is calling, applies limits, records usage, and can send the request to a different provider if the first fails. Your app talks to the gateway; the gateway talks to the model.

It is the same idea as an API gateway in web development, adapted to the traits of language models: long-running streamed responses, billing by token, and many interchangeable providers.

Following one request

Here is the path of a single chat message through a gateway.

1. Your app sends POST /v1/chat/completions to the gateway with a key the gateway issued.
2. The gateway looks the key up. It knows which team the key belongs to and what budget remains.
3. It checks rate limits for that key and rejects the request with HTTP 429 if the key is over its allowance.
4. It picks an upstream, such as a hosted API, and swaps in the upstream's own secret key.
5. It forwards the request, and streams the response back as it arrives.
6. When the stream ends, it reads the token counts, adds them to the spend record, and writes a log entry.

The app never sees the upstream secret. That separation is the reason many teams adopt a gateway in the first place.

What gateways typically do

Feature	What it means in practice
Authentication	Issue one virtual key per app or person, without handing out provider keys
Routing	Choose a model or provider by name, rule or cost
Fallback and retry	Try a second upstream when the first fails or is overloaded
Rate limits	Cap requests per minute and parallel requests per key
Budgets	Stop a key when it has spent a set amount
Logging	Record who called what, how long it took and how many tokens it used
Caching	Return a stored answer for an identical request
Guardrails	Filter or rewrite prompts and replies by policy

Not every gateway has every feature, and the exact set differs between products. Treat the table as a map of the usual territory.

The gateway is not the model

The most common confusion is thinking that a gateway provides answers. It does not. Behind it there must be a model endpoint: a service that holds the weights, runs the GPUs, and returns text. The gateway can only forward what that endpoint says.

That has practical consequences. Token prices are set by the endpoint. Content behaviour, such as whether lawful but mature fiction is answered or refused, is decided by the model, not the gateway (unless you add guardrails that make it stricter). The context window belongs to the model. This site's endpoint, for example, serves one open-weight text model with a 100,000-token context window, and charges $0.15 per 1M input tokens and $0.50 per 1M output tokens.

The endpoint often has gateway-like features of its own. This one returns X-Request-Id, X-Balance-USD and rate-limit headers, enforces 8 concurrent requests and 300 per minute on a full key, and lets you hold up to 5 labelled keys once you have topped up. A gateway in front of it adds a layer across providers, not a replacement for those basics.

A gateway's job in about twenty lines

To see what a gateway automates, here is a client-side sketch of two of its jobs: retrying when the upstream asks you to wait, and recording spend. A real gateway does this for every app at once.

Copyimport time
import requests

UPSTREAM = "https://api.llmgatewayopensource.com/v1/chat/completions"
spend = 0.0

def call(prompt, key, tries=3):
    global spend
    for _ in range(tries):
        r = requests.post(
            UPSTREAM,
            headers={"Authorization": "Bearer " + key},
            json={"model": "uncensored", "messages": [{"role": "user", "content": prompt}]},
            timeout=120,
        )
        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", "5")))
            continue
        r.raise_for_status()
        u = r.json()["usage"]
        spend += u["prompt_tokens"] / 1e6 * 0.15 + u["completion_tokens"] / 1e6 * 0.50
        return r.json()["choices"][0]["message"]["content"]
    raise RuntimeError("upstream stayed busy")

Multiply this by many apps, many keys and several upstreams, add a database and a dashboard, and you have the core of a gateway.

Streaming, tools and other tricky parts

Gateways look simple on a diagram and get complicated in details. If you evaluate one, check these points.

• Streaming. Chat replies arrive as server-sent events. The gateway must pass chunks through as they come, not wait for the full reply. It should also pick up the token counts reported at the end of the stream, which this API sends when you set stream: true.
• Tool calling. Tool definitions, tool_choice, streamed tool_calls and role: "tool" results all have to cross the gateway unchanged. A gateway that rewrites them can break agents in odd ways.
• Failover mid-stream. If an upstream fails halfway through a reply, the gateway cannot cleanly switch providers. Good ones fail over only before the first chunk.
• Concurrency slots. A streamed request holds its slot at the endpoint until the stream ends. A gateway that opens too many parallel streams will meet 429 responses with the code concurrency.
• Added latency. One more network hop costs time. Measure it on your own setup instead of trusting a generic number.
• Prompt storage. Check what is logged by default and how to turn it off.

If you are exploring the idea, you can test the endpoint side first. Calls to a gateway and to a model API use the same JSON, so practice on a model API now and put the gateway in front when you need it.

Do you need one?

Ask three questions.

• How many providers do you call? One provider and one app rarely justify a gateway. Several do.
• How many people or services need separate keys and budgets? If it is you alone, the provider's own key page is enough.
• Do you need an audit trail you control? If a policy demands that all prompts pass through your own logging, a gateway fits.

If the answers are "one, one, no", skip it and call the API directly. Create a key on the API key page, take the $0.50 trial for a spin, and read the docs for the supported routes. If the answers change later, the gateway can be added in front without touching your application logic, since both sides speak the OpenAI format.

Terms you will see

LLM proxy

Often a synonym for a gateway. Some tools use "proxy" for the simple forwarding layer and "gateway" for the version with keys, budgets and a dashboard.

AI gateway

A broader name that may include other model types such as image generation. This site covers text only.

Router

A component, or a service, that picks which model answers. A gateway may include one. The reasons developers pick a routing service are covered in the open router alternative article. This endpoint is not a router across vendors: it serves one model.

More guides

• LLM Gateway Open Source or Hosted API: How to Decide
• Why developers choose this open router alternative