For business
What it is
One front door to every AI model: it picks the right model for each task, switches to a backup if one fails, and tracks cost.
Why it matters
No lock-in to a single provider, outages do not stop the business, and simple tasks do not run on expensive models.
Risk without it
One provider’s outage or price change becomes yours, and costs stay invisible until the invoice.
For engineers
How it works
Every model call goes through one service that handles routing (a small model to classify, a large one to draft), retries and fallback, rate limits, prompt caching, streaming, and cost and latency logging per run.
Typical stack
- gateway service
- LiteLLM-style routing
- Anthropic / OpenAI / Bedrock APIs
Pitfalls & trade-offs
- Fallback models need their own evals: their output differs.
- Respect data-residency rules per provider.
- Log tokens and cost per tenant and per feature.
In this example
In
→ assembled prompt
Out
model: large fallback: ready streaming …
Fictional sample · not a client system