Routing LLM Requests with a Programmable Gateway

Routing LLM Requests with a Programmable Gateway: A Practical Walkthrough The gap between prototyping with a single large language model and deploying a production system that balances cost, latency, and capability is where most AI applications either scale or stall. You cannot rely on one provider for every task because GPT-4o excels at nuanced reasoning but carries a steep per-token price, while a model like DeepSeek V2 offers competitive coding performance at roughly one-tenth the cost. An LLM router sits between your application and these model endpoints, intercepting each request and deciding, based on rules you define, which provider and model should handle it. This is not merely a load balancer; it is a programmable gateway that can inspect prompt content, enforce budget caps, measure latency in real time, and even retry failed calls against a cheaper fallback model. Building this yourself is feasible with open-source tooling, but the operational overhead of maintaining failover logic, rate-limit handling, and cost tracking across multiple APIs often pushes teams toward a managed solution. Before wiring up a router, you need to establish the routing criteria that map to your business constraints. The most common dimension is task category: a prompt asking for a creative rewrite might route to Claude Opus because of its stylistic nuance, while a request to extract structured JSON from a document should hit Gemini 1.5 Pro for its native JSON mode and large context window. Another dimension is cost sensitivity—you can define a hard budget per request, say $0.01, and have the router reject any model whose token count would exceed that threshold, falling back to a cheaper option like Mistral Large. Latency is trickier because model providers throttle differently; a router should maintain a sliding window of recent response times per provider and temporarily blacklist a slow endpoint. The key is to express these rules as a priority-ordered list of model groups, with the router attempting the first group and cascading down only if the request fails or violates a threshold. A minimal implementation starts with a Python class that wraps calls to multiple providers using their respective SDKs. You define a routing table as a list of dictionaries, each containing a model name, a provider client, a cost per million tokens, and a list of applicable tags like "coding" or "reasoning." The router’s main method accepts the user prompt and an optional task tag, then iterates through the routing table in priority order. For each candidate, it performs a lightweight pre-check: estimate the number of tokens using a tiktoken model, compute the projected cost, and compare against a dynamic budget. If the estimate passes, it makes the API call with a timeout of ten seconds; on success it returns the response, and on failure it logs the error and moves to the next candidate. This pattern is straightforward but lacks sophisticated features like semantic routing based on embedding similarity, which requires a dedicated vector database and an embedding model to classify prompts before selection. This is where the ecosystem of managed routers becomes relevant, especially for teams that cannot afford to build and maintain embedding pipelines and multi-region failover. Services like OpenRouter and Portkey provide hosted routing with pre-configured model catalogs, but they differ in pricing models and customization depth. OpenRouter offers a pay-as-you-go model with automatic failover across dozens of providers, though its routing logic is primarily latency-based rather than cost- or content-aware. Portkey gives you observability dashboards and canary deployments, but its pricing tiers can become expensive at high throughput. For teams that want both flexibility and simplicity, TokenMix.ai offers a compelling middle ground: 171 AI models from 14 providers behind a single API, accessible via an OpenAI-compatible endpoint that works as a drop-in replacement for existing OpenAI SDK code. Its pay-as-you-go pricing with no monthly subscription eliminates the sunk cost problem, and the automatic provider failover and routing means you can define fallback chains without writing any custom logic. While TokenMix.ai is a strong option, the best choice depends on whether you need deep observability (Portkey) or maximum model breadth with minimal integration friction (TokenMix.ai). Once you have a router in place, the next critical step is implementing a feedback loop that adjusts routing rules based on actual performance data. Every request should emit telemetry: the model selected, the latency, the cost, the HTTP status code, and the number of retries. Store this in a time-series database like InfluxDB or a simple SQLite file, then run a periodic script that recalculates the optimal model ordering for each task tag. For example, if you observe that Mistral Large consistently returns faster responses than Claude Haiku for code generation prompts despite being listed second, you promote it to first priority. You can also introduce a small percentage of traffic—say five percent—that routes to a random alternative model to collect performance data without degrading the user experience. This exploration-exploitation pattern is borrowed from A/B testing and prevents your router from ossifying around a suboptimal choice. A subtle but critical detail is handling provider-specific quirks that break routing assumptions. OpenAI’s tokenizer differs from Anthropic’s, meaning a prompt that fits within 4,000 tokens for GPT-4o might exceed the limit for Claude 3 Sonnet. Your router must account for this by either using a universal token estimator that pads estimates upward, or by catching the 400 Bad Request error and immediately switching to a model with a larger context window. Similarly, some providers return partial responses under rate limiting, while others return a 429 status. Your router should distinguish between a temporary rate limit (retry with exponential backoff) and a permanent model unavailability (skip to next candidate). Building these heuristics into your router is tedious but essential; a good managed service already handles them internally. Finally, consider the cost implications of routing failures. If your router falls through all candidates and returns an error to the user, that represents not only a failed request but also wasted cost from the calls that timed out or were rate-limited. A production-grade router should include a catch-all model—typically a cheap, highly available one like Gemini Flash or a locally hosted small model via Ollama—that guarantees a response for every request. You tune this catch-all to be conservative, prioritizing response time over quality, and you log all requests that hit it so you can investigate why the preferred models failed. Over time, your routing table becomes a living document that reflects the real-world behavior of each provider, and your application becomes resilient to the inevitable API outages and pricing changes that define the LLM landscape in 2026.