Routing

Which model answers, whose key pays, and what happens when it fails. Part built, part designed; each section says which.

Replaced in part: models.md moved routes, their credentials and the models they name into provider accounts, models and chains. Traffic classes, the breakers and attribution below carry over, and where this says "route" or "row" it now means a link in a class's chain; traffic_routes itself is no longer read.

Who this serves

An operator runs outturn. Workspaces are the operator's customers, and workspaces have customers of their own. The operator's business decides who owns the model relationship, and three arrangements have to work on one gateway:

1. Operator-provided. The operator holds every key, routes as it likes for cost and reliability, and bills workspaces for usage or folds it into the price. A freight brokerage platform whose quoting is certified against one model works this way.
2. Workspace brings a key. The workspace pays their provider directly and pays the operator only for hosting. A customer-service workspace with its own Anthropic account.
3. Operator runs its own inference. A vLLM serving an open model, sold at a margin, with a hosted model behind it for when the vLLM is full.

Fallback has to work across all three, and the bill has to say afterwards which one paid for which call.

Traffic classes, not model names

Built. Nothing above the gateway names a model. An agent's policy names a traffic type -- assistant today -- and the gateway resolves it through its chain to an ordered list of attempts: protocol, base URL, model, credential. A skill will name a class the same way. The workspace and the workspace's customers never learn which model served them unless the workspace is the one holding the key.

assistant is the generic class: a person talking to an agent. background is the same work with nobody waiting on it -- a turn a schedule or a webhook started, which carries no user -- so an operator can send it to a cheaper or slower model with a row, and the ledger can tell what people asked for from what ran on its own. It is the same judgment that queues such a turn behind anybody who is waiting. A class the agent's policy names wins over it: an agent confined to one class is confined for a reason, and a schedule is not a way round that. session-name is the third: the one request made after a conversation's first turn to give it a title, sent by the API tier under its own class so an operator can point it at a small model with a row. More come as they are needed -- customer-service, summarize, compaction -- and each is a row set, not code.

Absent a chain for a class, it is refused, with a message saying where to give it one: the gateway keeps no provider of its own to fall back to, since one nobody chose would answer every unconfigured class silently. Two classes borrow instead. background and session-name walk assistant's chain until they have their own, because a deployment that said where its assistant traffic goes has said where a scheduled turn or a title should go too. vision never borrows: its model has to be one that can see.

Absent is not the same as rejected. A class whose rows all exist and are all turned down -- every base URL pointing inside the network with no operator having opened that host, or no credential held for any of them -- is refused rather than served by anything else. A route was configured and declined, and anything else would be a different vendor with a different model: a deployment confined to an in-cluster provider would otherwise begin sending its prompts to a public API without saying so, and a momentarily missing credential would silently downgrade the model mid-conversation. Refusing is what somebody notices; the fix for an in-cluster route being skipped is to name its host in OUTTURN_INTERNAL_HOSTS.

A class may carry a default priority for jobs that arrive without one. It is a fallback, not a derivation: priority is a property of the trigger (is somebody waiting?), class is a property of the task (what is good enough?), and the same class runs at both. A message handler enqueues realtime, a scheduler or webhook enqueues background, and only a job that says nothing takes the class's default. summarize and compaction default to background because nobody is ever watching them.

Rows as the whole of routing

Built: a chain per class, the operator's by default, which a workspace's own would replace once a workspace can have one; its links tried in order, skipping any whose circuit is open. traffic_routes was where this lived first and is kept, unread, for a release.

Designed, in part built: two more facts about each link.

Credential owner. Today every key is the operator's: a provider account's key is a variable in the gateway's environment or a sealed provider key the operator typed into the Models page (models.md). A row will say whether its key is operator-held (an env var, as now) or workspace-held (a key the workspace entered, stored by the gateway encrypted under a master key from its own environment). The gateway is already the only tier that holds credentials, so this widens nothing; the API proxies the workspace's submission and learns only that a key is on file and its last few characters.

A workspace's list can then mix freely: Bedrock with their AWS credentials, then Anthropic direct with their own key, then -- if offered -- the operator's model. Same-model fallback (Bedrock to Anthropic direct) needs nothing more than two rows.

Fallback kind. Whether stepping to this row keeps the model or changes it. Same weights through a different door is safe to fall through silently. A different family is allowed but visible: recorded on the reply and the ledger as a model change, shown in the UI, and something a workspace opts into per route rather than gets for free. Cross-provider failover is not honest until the durable transcript is separated from the per-provider projection (see compaction.md), because a transcript that has been through two families holds artifacts each rejects. A skill certified on one model declares that it accepts same-model fallback only; the workspace cannot loosen that, the operator can.

The editor infers the kind so the obvious cases need no labeling. Built so far: the chain editor marks a link to a different family, and the gateway records a change of model against the reply it answered, shown under it as "Agent model changed" (models.md); a workspace opting in waits on a workspace having chains of its own.

The surcharge fallback

Designed. A workspace with their own key may opt into "during provider outages, fall back to the operator's model; additional charges apply." It is just the operator's rows appended after the workspace's, and one fact on the bill: those calls were paid with the operator's credential.

• Off by default, per workspace.
• The operator decides whether to offer it at all. A hosting-only workspace has no usage relationship to charge against and never sees the checkbox.
• The breaker decides when it engages, not the workspace. The workspace's rows are always tried first, so nobody routes to the operator's model by preference.
• Fallback is per model call. A turn that fails mid-stream on the workspace's key is retried, and the retry may land on the operator's model. The transcript, not the session, is the unit of billing, and the ledger says which call went where.

Two breakers, not one

Built: a breaker shared across gateway pods through Postgres, keyed by endpoint (protocol and base URL), ignoring client errors so a rejected request or a rate limit does not mark a provider dead.

Designed: split it, because two different things trip it once workspaces hold keys.

• Endpoint health stays cross-workspace. Connection refused, timeouts, 5xx. A dead provider is dead for everyone and one pod learning it should spare the rest. Keyed by endpoint, as now. An overloaded vLLM is endpoint-wide and belongs here.
• Credential health is per credential. 401 and 403 mean this key is bad; 429 means this key is throttled. Neither says anything about the endpoint or about anyone else's key. Keyed by credential, a short cooldown for 429 and a longer one for 401, never propagated to the endpoint breaker. A workspace's expired key must not take a provider down for the other workspaces.

The walk skips a row if either breaker is open, and the ledger row that eventually serves records which one it was. "Why did we fall back at 3pm" is answered with "your key was rate-limited", not "Anthropic was down", which matters when the surcharge is on the bill.

The walk goes on to the next row only when the request was not taken: the provider answered with a failure, asked to be retried, or was never reached at all (a refused connection, a name that did not resolve). A request that was sent and whose answer was then lost -- a read that timed out, a body cut off, a reply that would not parse -- ends the walk instead. The provider may have done the work and billed for it, and the next row would do it again; the gateway logs the attempt with its endpoint, model and workspace, since nothing else records it.

Today a 429 does not count at all, so a workspace hammering their own key fails the row and moves on every call. With a per-credential breaker the row is skipped for the cooldown instead, which is cheaper and is the backoff the provider asked for.

Caching

Which parts of a prompt a provider is asked to cache is in caching.md. Per provider, since only Anthropic needs asking.

Attribution

Every call is a row in the usage ledger (see usage.md) naming the endpoint, the model actually served, the credential owner, and the fallback kind if it was one. The bill hangs off that, not off routing configuration, because configuration changes and the ledger does not.

Order of work

The ledger first: routing changes are what produce multi-provider turns, and the current per-message usage summary is wrong for those. Then credential owner and fallback kind on the rows, workspace-held keys in the gateway, the credential breaker beside the endpoint one, and the surcharge checkbox. Then more traffic classes as skills need them.