LLM Gateway: What Product Managers Need to Know

What Is an LLM Gateway and Why Did It Emerge?

In 2023, most teams called one model from one provider. The architecture was simple: application code hits the OpenAI API directly. That worked at small scale.

By 2025, the picture had changed. Teams were routing traffic across four or five providers depending on cost, capability, and latency requirements. Different teams inside the same company were making independent API calls, each with their own API keys and no shared visibility into total spend. A single provider outage could take the product offline with no automatic fallback.

The LLM gateway emerged to solve these coordination problems. At its core it is a reverse proxy for model API calls: your application sends every inference request to the gateway, and the gateway handles routing, authentication, rate limiting, logging, and fallback before forwarding to the actual model provider.

The gateway sits between your app and every model it calls

Your Appsends request toLLM Gatewayroutes toProvider

The gateway applies policy rules (rate limits, cost budgets, routing logic) before every call, and logs every response for observability.

Without a gateway, each of these concerns gets solved in application code, duplicated across teams, and often inconsistently enforced. The gateway makes them infrastructure: configured once, applied everywhere.

Core Features Every Gateway Provides

Not all gateways are equal, but any solution worth using covers at least these five areas. As a PM, these are the capabilities you need to understand when evaluating options or working with engineering on a custom build.

Centralized Authentication

One API key management layer instead of individual keys per team. The gateway holds provider credentials and exposes internal API keys to your application teams. When a key needs rotating, you rotate it in one place. When a team's access should be revoked, you update the gateway, not every app that team touches.

Request Routing and Model Selection

Rules that direct traffic to different models based on request properties, user tier, or real-time cost and latency signals. A production gateway might route a simple classification request to a small model at $0.15 per million tokens and a complex reasoning request to a frontier model at $15 per million tokens, automatically, without application code knowing the difference.

Rate Limiting and Quota Management

Per-team, per-feature, and per-user rate limits enforced before requests leave your infrastructure. If a batch job tries to consume your entire hourly quota at 2am, the gateway throttles it so your user-facing product still has capacity. This is the PM decision: who gets priority when supply is constrained.

Cost Tracking and Attribution

Every request tagged with metadata (feature, user, team, environment) and its token cost logged. The gateway makes cost attribution possible at the granularity that actually drives decisions: by feature, not just by model. Without this tagging layer, cost reports show you total spend by provider and nothing more useful.

Provider Fallback and Redundancy

When a primary provider returns an error or exceeds latency thresholds, the gateway automatically retries on a secondary provider. Users see a slightly slower response instead of an error page. The fallback policy is a product decision: which provider can substitute for which, and at what quality tradeoff.

Some gateways also provide prompt caching passthrough, response caching, PII scrubbing, and semantic routing. These are useful but not universal. Evaluate them against your specific requirements rather than treating them as table stakes.

Build vs. Buy: When to Use an Existing Solution

The gateway decision follows the same build vs. buy logic as any infrastructure component. The answer depends on how differentiated the capability is for your specific product.

Buy (use an open-source or managed gateway)

You call two or more providers today
Multiple teams make independent LLM API calls
You have no visibility into per-feature cost
A provider outage would take your product offline
You want routing logic without application code changes

Build (custom gateway layer)

Your routing logic has proprietary business rules that external tools cannot express
You have extreme latency requirements where gateway overhead matters
Regulatory constraints prohibit third-party infrastructure touching your requests
You are already operating a service mesh and the gateway is one more node

The most common mistake is building when you should buy. Teams underestimate the ongoing maintenance cost of a custom gateway: provider SDK updates, new authentication flows, monitoring, and the engineering time to add features that commercial gateways ship by default. Start with an open-source option and graduate to custom only when you have hit the ceiling of what existing tools can express.

Commonly evaluated options (as of Sept 2026)

LiteLLM: Open-source, supports 100+ models, strong cost tracking, self-hosted or managed.
Portkey AI: Managed, built-in semantic caching, strong observability dashboard.
Kong AI Gateway: Enterprise-focused, integrates with existing Kong deployments, strong rate limiting primitives.
Cloudflare AI Gateway: Edge-native, minimal setup, good for globally distributed products but limited routing logic.

Feature sets change quickly. Always test against your actual provider mix and routing requirements before committing.

Build AI Infrastructure Fluency in the Masterclass

The AI PM Masterclass covers how to reason about infrastructure decisions like gateway architecture, taught live by a former Apple and Salesforce Sr. Director PM.

How to Evaluate Gateway Options

Gateway evaluation has four dimensions that matter for product teams. Weight them based on where you are today, not where you want to be in two years.

Provider coverage

Does the gateway support every model you use today and the ones on your roadmap? Verify native support (not just a passthrough wrapper) for your primary providers. A gateway that requires a custom adapter for every new provider adds engineering overhead that negates the point.

Routing expressiveness

Can you express the routing rules your product requires? Simple round-robin is easy. Rules that consider user tier, request complexity score, current provider latency, and fallback priority are harder. Get specific about your routing requirements before evaluating, then test them against each candidate.

Observability output

What does the gateway export and in what format? You need per-request logs with token counts, latency, model, cost, and the metadata tags you define. Evaluate whether the output integrates with your existing observability stack (Datadog, Grafana, etc.) or requires a separate dashboard.

Latency overhead

The gateway adds a network hop. For user-facing features where time to first token matters, measure actual overhead in your infrastructure, not vendor-quoted numbers. A 30ms gateway overhead is acceptable on a 3 second generation. It is not acceptable on a 200ms autocomplete endpoint.

Operational burden

Who maintains the gateway and how? Self-hosted open-source has no licensing cost but requires engineering time for upgrades and incident response. Managed services charge for uptime guarantees. Factor in your team's actual capacity to own another piece of infrastructure.

The Product Decisions That Belong to You

Engineering can implement the gateway. But several decisions require product input because they reflect business policy, not technical preference. Make sure you have a clear opinion on each before the implementation begins.

Priority tiers under contention

When the gateway must throttle requests, which features get priority? Typically: user-facing over background, paying tiers over free, synchronous over async. Define the policy explicitly or engineering will make arbitrary calls.

Quality floor for fallback providers

Not every provider can substitute for every model. A fallback to a lower-capability model is sometimes acceptable (classification tasks) and sometimes unacceptable (legal document review). Document acceptable fallback paths per feature, not per product.

Cost budget per feature

What is the maximum monthly spend allowed for each AI-powered feature? The gateway can enforce these budgets, but someone has to set them. This is a product and finance decision encoded into infrastructure.

PII and data residency rules

Which request types require that data never leave a specific geographic region? Which features are prohibited from sending certain data fields to external providers? These rules must be configured in the gateway, which means they must be defined first.

Logging retention and access controls

Gateway logs contain prompt and completion content. Who is allowed to access these logs, for how long, and for what purpose? This is a privacy and compliance decision that shapes the gateway's log configuration.

Semantic caching eligibility

Serving a cached response to a similar but not identical query is faster and cheaper, but may not be appropriate for real-time data queries or highly personalized outputs. Decide which request types are cache-eligible based on acceptable staleness for each use case.

Implementation Gotchas to Brief Your Engineering Team On

Even well-planned gateway implementations hit predictable problems. Flag these early so they do not become surprises in production.

Streaming responses require gateway-level support

If your product uses streaming (tokens appear as they are generated), the gateway must support streaming proxy mode. Many early gateway implementations buffered the full response before forwarding it, destroying the streaming UX. Verify streaming support explicitly with a test before committing.

Context window and timeout mismatches

Different providers have different context window sizes and timeout behaviors. A request that succeeds on Provider A may fail on Provider B if the fallback provider has a smaller context window. Your fallback routing must account for context size constraints, not just capability labels.

Metadata tagging is a discipline problem, not a technical one

The gateway can only attribute costs to features if every request includes the right tags. This requires every team that calls the gateway to follow the same tagging convention. Establish and enforce the schema before onboarding multiple teams, not after. Retroactive attribution is difficult.

Gateway outages have blast radius equal to your entire AI product

If the gateway is unavailable, every feature that calls an LLM through it is unavailable. Design for gateway high availability from the start: multi-zone deployment, health checks with fast failover, and a circuit breaker that falls back to direct provider calls for critical paths during gateway incidents.