Technology
Tokens are sold like a commodity. They aren’t delivered like one.
Harry Yu & Jeffery Wong
Pick a few suppliers, compare the price per 1M tokens, done. That's how we bought LLM access when we started building AI agents that handle customer experience for enterprise clients, and it's how most of the industry still buys it.
Millions of requests later, we know where that breaks. What you are really buying is a model's behavior at a given endpoint, and that behavior varies in ways you can't see until you run at volume. Here is what we found, and what we built to make tokens behave like the commodity they are sold as.
Supply: the same model is not the same product
Everyone who runs on LLM APIs knows the obvious failures: outages, rate limits, a model retired on the vendor's timetable. We hit all of those. What we did not expect is how often a vendor returns a healthy 200 that is not the product we paid for.
Same model, different price plan, different answer. When quota errors rose on pay-as-you-go while our traffic stayed flat, we bought reserved capacity for the same model: same workspace, same key, same name with a suffix. The reserved endpoint ordered its answers differently, putting the decision before the reasoning, and on hard requests the reasoning ran until the output was cut off. Every call returned HTTP 200, identical on replay, with nothing to trigger a failover. We rolled back within two days.
Same model, not wrong, just slow. A model that normally answers in well under a second took several times as long, and every response was a healthy 200. For a voice agent that is worse than an outage. A fast failure trips the fallback in milliseconds and the caller never notices. A slow success is never retried, so every extra second lands on the customer as silence.
Same model, same endpoint, changed underneath. The same deployment, called with the same request, started answering differently, and nothing on the status page or in the deployment name said why. We only knew because every response carries a system fingerprint, an opaque hash of the backend that produced it, and we log it. One day that fingerprint changed. The model behind the deployment had been swapped. Without that one logged field we would have been debugging our own prompts.
None of these looked like failures. Every one returned a healthy 200, and every one would have passed a quick test call. That is the part people underestimate. The supply problem is not outages. It is the same model turning out to be a different product depending on which endpoint serves it, and when.
Demand: what we need tokens to be
Supply is half the problem. The other half is that we never needed one kind of token. Our agents work across different modalities and time horizons, from live voice and chat, emails, offline analysis, and scheduled jobs. Each has different requirements for speed, accuracy, and cost, and often needs a different model or vendor. Yet we need the same fallback and compliance guarantees across all of them.
No single supplier wins that table. The model that is fastest for a voice turn is rarely the most accurate for offline analysis, and the cheapest one for scheduled tasks is the wrong one for an email a customer will keep. Each row has its own best answer, and the answer changes every time a vendor ships, retires or re-prices a model. What we want is a layer that offers those options, so our agentic orchestrator can pick the model each task needs without carrying the supplier logic itself.
We could have put an aggregator between us and the suppliers. It would have added a hop of latency, stacked its own outages and rate limits on top of the vendor’s, and hidden the routing behind logic we cannot see [1].
So we built an LLM gateway: one front door that every agent calls and every model is bought through. It began as a router choosing which supplier gets a request, and became a gateway once it also carried compliance, cost and limits.
What we built: one provider for every model
The gateway presents every model, from every vendor, for every capability, as one provider with one contract. Callers send the OpenAI request shape and get the OpenAI response shape back, whichever vendor answered. Behind it each vendor keeps its own API, protocol and quirks, and the gateway absorbs the difference. Switching a model is a change to one string. Switching a vendor is a change to one config file.
Every request takes the same path. Identify the caller. Apply its rate limit. Resolve the route against the environment’s allowlist and residency policy. Sample it for shadow testing if a trial is running. Walk the fallback chain under a timeout. Adapt the parameters for whichever provider is being tried. Send, shape the response, and leave a trace of which route answered and every provider tried along the way.
Everything the gateway does, failover, shadow testing, rate limits, tracing, is a stage on that one path rather than a separate system. It is also what lets our agent orchestrator get simpler. The orchestrator no longer knows which vendor is up, which region a model lives in, or which parameter a model refuses. It asks for a capability and a model, and the gateway does the rest.
What it buys us
We can sleep. Every model has a fallback chain and every provider a circuit breaker shared across all pods, so an error or a timeout sends the request to the next provider in the chain, and a provider that keeps failing is skipped until it recovers. A scheduled probe calls every chain so a dead backup is found before a real request needs it. The failures that return a 200 are caught by the trace on every response: fingerprint, finish reason, token counts, cache hit rate, latency, field shape. A model swap or a truncation shows up as an anomaly within minutes, and moving off the endpoint is one config line no caller sees. The one that still keeps us up is the slow 200. A timeout turns a slow success into a fast failure that fails over, but a response that is slow and inside the timeout is served. That is why latency-based routing is next.
New models are measured before they're trusted. A new OpenAI release, a decision model we have not used before, a new text-to-speech voice: the questions are always the same. How is the latency, does it stream, what does it cost. A new model is a config entry in the gateway, and from that moment it appears in a live catalogue with a built-in test bench where anyone on the team can try it on their own prompt and see time to first token, where the time went, cost and which route answered, or put it under load and measure throughput before it is trusted with traffic. A model stops being a name on a vendor page and becomes a number you measured yourself.
Tested on live traffic before it ships. Offline evaluated results do not always hold on live traffic, so before it carries any, it is shadow tested: the caller is answered by the live model as usual, and a sampled share of the same requests goes to the candidate in the background. A judge compares the two answers, so the decision to switch rests on how the candidate handled our real traffic, not on a public benchmark. Offline and online have to agree.
Built once, every agent gets it. Because every call passes through one place, anything built there is built once and applies to every agent. Every call is priced and attributed to a client. Each client has its own request budget, so one client's burst cannot consume another's supply, and the limiter can never cause an outage on its own. Monitoring, alerts and vendor moderation verdicts come from the same place. None of this needed a change in any agent.
Our own GPUs are just another supplier. Alongside the vendors we run our own fine-tuned models on our own GPU nodes. Each one is registered like any other provider, with the same breaker, fallback chain, trace and cost attribution, so the orchestrator can route to it as freely as to any vendor, with the same guarantees. We do this where a small model trained on our own data beats a vendor on cost and latency for its task.
The road ahead
Route on latency and task priority, not only on errors. Today a provider is failed over on an error or a timeout, so a slow 200 inside the timeout is served, and that is the failure that hurts most. Next, latency becomes a health signal: a model that drifts past its own baseline is routed around before the timeout, the same way a 429 is. From the same lesson, queuing by task importance, so a live conversation is served before a scheduled job and an offline analysis never touches the supply the voice and chat agents need.
More of our own models. We already run a few fine-tuned models of our own, and we want more: one for each use case where the task is narrow, the volume is high and we hold a lot of our own labelled data. The decision models we recently introduced are the latest example, and we wrote about where that kind of model fits between LLMs and purpose-trained classifiers. Each new one goes through the same path as any vendor model: offline evaluation, shadow testing, promotion.
Tokens are not a commodity. The same name is not the same product, the supply moves on the vendor's schedule, and the failures that hurt most return a healthy 200. With a gateway in front of them, they become one: swappable, measured and under our control. The gateway's job is to make the supply boring, so that the agents on top of it can be interesting.
References
[1] So you want to use OpenRouter, DeepSeek V4 Flash 0731 on TAU-Bench Airline, 2026-09-07: first-party 81.3%, DigitalOcean 58.4%.
