LLM Gateway Open Source or Hosted API: How to Decide

What you are really choosing between

An llm gateway open source project is software you run yourself. It accepts OpenAI-style requests from your apps and forwards them to one or more model providers, adding keys, budgets, logging and routing on the way. A hosted API is the other end of that pipe: a service that actually runs the model and bills you per token.

So the real comparison is "gateway plus provider" against "provider only". For one model and a small team, provider only is usually enough. For several providers, several teams or strict spending rules, a gateway starts to earn its keep.

What self-hosting a gateway involves

Open source gateways are well known. LiteLLM's proxy is one example; others exist. Whichever you pick, the work is similar.

Things you now run

• A service that must stay up, because every app depends on it. It is a single point of failure unless you run more than one copy.
• A database if you want virtual keys, spend tracking and per-user budgets.
• Upstream secrets for each provider, stored safely and rotated when needed.
• Logging and retention. Logs of prompts are sensitive. You decide what to store and for how long.
• Upgrades. Provider APIs change, and so does the gateway. Expect to follow releases.

Things that stay the same

The gateway does not change your token price. It also does not add models. If your upstream refuses a prompt, the gateway returns the refusal. Its value lies in control and visibility, not in cheaper or looser answers.

What calling a hosted API directly gives you

A hosted OpenAI-compatible endpoint removes the middle layer. Your app talks to the provider with a base URL and a key.

• No service to deploy, no database, no upgrade cycle.
• Prepaid billing with a visible balance. Here that is $0.15 per 1M input tokens and $0.50 per 1M output tokens, and the remaining credit comes back in the X-Balance-USD header.
• Published limits: 8 concurrent requests and 300 per minute on a full key.
• Key management on the provider's side. You can regenerate a key at any time, and accounts that have topped up can hold up to 5 labelled keys and revoke any of them.

What you give up is a single place to apply your own policy across providers. If you only use one, that loss is small.

A decision table

If you have...	Lean towards
One model, one app, one or two developers	Hosted API, directly
Several providers and a need for fallback	Gateway
Many teams, each needing its own budget and key	Gateway
A rule that prompts must never leave your network	Self-hosted model, with or without a gateway
No one to carry a pager for infrastructure	Hosted API, directly

Cost is the other axis. A gateway is free software, but not free to run: add server hours, a database and the time spent maintaining it. If those cost you more than the budget controls save, skip it for now. You can add one later without rewriting anything, because both sides speak the same OpenAI format. The article on why developers choose this OpenRouter alternative covers the routing-service angle.

Putting this API behind a gateway

If you later adopt a gateway, this endpoint is just another OpenAI-compatible upstream. With LiteLLM's proxy, a model entry looks like this:

Copycat > config.yaml <<'EOF'
model_list:
  - model_name: uncensored
    litellm_params:
      model: openai/uncensored
      api_base: https://api.llmgatewayopensource.com/v1
      api_key: os.environ/UPSTREAM_KEY
EOF
export UPSTREAM_KEY="paste-your-key-here"
litellm --config config.yaml

Your apps then call the gateway instead of the provider. Only the base URL and key change:

Copyfrom openai import OpenAI

# Direct to the hosted API. Through a local gateway you would use
# base_url="http://localhost:4000/v1" and a key issued by the gateway.
client = OpenAI(base_url="https://api.llmgatewayopensource.com/v1", api_key="YOUR_KEY")

r = client.chat.completions.create(
    model="uncensored",
    messages=[{"role": "user", "content": "Give me three names for a fantasy tavern."}],
)
print(r.choices[0].message.content)

Set the gateway's per-model limits to match the upstream: no more than 8 parallel requests and 300 per minute for a full key. Past that the API returns HTTP 429, with the code concurrency or rate_limited, and on temporary overload a Retry-After: 5 header. A gateway should retry after that delay instead of passing the error straight to users. Remember that a streamed request holds its concurrency slot until the stream ends.

Security and reliability of the extra layer

A gateway concentrates risk. Everything passes through it, which makes it both useful and sensitive.

Security questions

• Who can open the admin interface, and is it exposed to the internet?
• Where do upstream keys live, and who can read them?
• Does the gateway store full prompts and replies? If so, for how long, and who has access? For private or creative text, a log of everything is a liability you are choosing to create.
• How quickly can you rotate a key if it leaks?

With a direct connection, much of this is handled on the provider's side. Prompts sent to this API are not used for training. You still own the key hygiene on your side.

Reliability arithmetic

Two services in a row are less available than either alone. As a hypothetical example, if your gateway and the upstream each work 99.5% of the time, a request succeeds about 0.995 x 0.995 = 99.0% of the time. We publish no uptime figures here, so measure your own. The point is that a gateway adds a link to the chain, which pays off only when its fallback logic recovers from more failures than it introduces.

The honest cost line

Count the hours. Deployment, upgrades, a dashboard nobody checks, a debugging session when streaming breaks in the middle of a reply. For a small team that time often exceeds the savings from budget controls. For a large team with many keys, the same time is cheap compared with chasing unexplained spend.

Start small, then add layers

1. Begin with a direct connection. Open the API key page, claim the $0.50 trial and run one call.
2. Log token usage yourself from the usage field. That is enough observability for a long time.
3. Add a gateway only when a concrete need appears: a second provider, a team budget, or an audit requirement.

The service is text only, offers one open-weight model run by an independent provider, and has a 100,000-token context window. It is not a router across vendors, so if you need that, a gateway or a routing service is the right addition. The docs list the supported routes: chat completions, responses and models.

More guides

• What Is an LLM Gateway? A Plain Explanation
• Why developers choose this open router alternative