Route first, answer second: how one app talks to 20+ AI models

When I launched the first version of ORA in 2024 it was a photo app. Custom PyTorch pipelines, a couple of GANs and Stable Diffusion, all for editing pictures. Two years and three versions later it answers questions, writes, codes, transcribes, generates images, video, music and voices, and it does that with more than twenty hosted models behind one interface.

The app did not get there by bolting on a model at a time. It got there because version 2 moved one decision out of the client and into the backend: which model handles this request. This post is about that decision, the small amount of code it took, and the mistakes I made before it.

The problem with "pick a model" menus

Most multi-model apps show a dropdown. Pick GPT, pick DeepSeek, pick LLaMA, type. That works for people who follow model releases. Everyone else sees a list of names that mean nothing and picks the first one.

It also leaks every change into the app. A new model means a new app build, a store review, and a week of users on old versions sending requests to something you have already replaced. With 20,000 users across iOS, Android, macOS and Vision Pro, that week is real.

So the rule for version 2 was simple. The client never names a model. It names a task.

The shape of a request

Every request from the app carries three things: a task, the content, and a few hints.

{
  "task": "chat",
  "hints": { "needsReasoning": true, "hasImage": false, "maxLatencyMs": 8000 },
  "messages": [{ "role": "user", "content": "Explain why my Dart future never completes." }]
}

The task is a short list the app owns: chat, code, write, image.generate, image.edit, video, music, voice, transcribe, ocr. The hints are things the client genuinely knows, such as whether the user attached an image or chose a "think longer" toggle. Nothing in the payload says which model to use.

The registry

The router is a NestJS service. At its heart is a registry: one record per model, loaded from configuration at startup and refreshable without a deploy.

type ModelEntry = {
  id: string;                 // "deepseek-r1", "llama-4-maverick", "kimi-k2", ...
  provider: "openai" | "deepseek" | "together" | "replicate" | "inhouse";
  tasks: Task[];              // what it is allowed to do
  traits: { reasoning?: boolean; vision?: boolean; longContext?: boolean };
  costPer1kTokens: number;    // or per image / per second of video
  p50LatencyMs: number;       // measured, updated from real traffic
  weight: number;             // preference among equals, 0 to 1
  enabled: boolean;
};

Two fields do most of the work. tasks says what a model is for, and traits says what it is good at. A request for chat with needsReasoning: true filters the registry down to enabled models that list chat and carry the reasoning trait. Everything after that is tie-breaking.

The picking function is short and I have resisted making it clever:

function pick(req: RouteRequest, registry: ModelEntry[]): ModelEntry[] {
  const fits = registry
    .filter((m) => m.enabled && m.tasks.includes(req.task))
    .filter((m) => !req.hints.needsReasoning || m.traits.reasoning)
    .filter((m) => !req.hints.hasImage || m.traits.vision)
    .filter((m) => !req.hints.maxLatencyMs || m.p50LatencyMs <= req.hints.maxLatencyMs);

  // Cheapest first among models with a decent weight; the rest become the fallback chain.
  return fits.sort((a, b) => b.weight - a.weight || a.costPer1kTokens - b.costPer1kTokens);
}

It returns a list, not a winner. The first entry gets the request. If it times out, rate-limits or returns an error the router moves to the second, and the user sees one answer and no error. That fallback chain was the single biggest improvement in perceived reliability, more than any retry logic I wrote before it.

What "good at" means in practice

The traits started as my opinions. They are now mostly measurements.

• Latency is sampled from real traffic per model and written back into the registry every few minutes. A model that is slow today is simply not chosen for requests with a latency hint.
• Health is a rolling error rate. Three failures in a short window disables a model for a cooling-off period. It comes back on its own.
• Cost is the published price. I keep a per-user daily budget on the router, so a free user asking a reasoning question gets a reasoning model, just not the most expensive one, and a subscriber gets the better one.

The result is that "which model" is answered by the system, continuously, and I stopped being the person who had to notice a provider was having a bad afternoon.

Adding a model without touching the app

This is the part I was after. Adding Kimi K2 to ORA took three steps and none of them involved Xcode:

1. Write a provider adapter if the provider is new. Most are OpenAI-compatible now, so this is often nothing.
2. Add a registry entry with its tasks and traits, enabled: false.
3. Flip it on for a few per cent of traffic, watch the latency and error rate the router is already collecting, then raise the weight.

Users on a build from six months ago got the new model the same afternoon. That is the whole argument for routing in the backend.

The mistakes

I let the client choose first. Version 1 had a model picker. Users chose by name, usually the one they had heard of, and then wrote reviews saying the app was slow. The app was not slow. The model they picked was.

I under-specified tasks. For a while image was one task. Generation and editing need different models with different inputs, and a single task forced ugly branching inside the adapters. Splitting it into image.generate and image.edit removed more code than it added.

I trusted provider status pages. They lag. A rolling error rate measured from your own traffic is the only health signal that is on time.

I forgot that fallbacks change the answer. If the primary reasoning model fails mid-conversation and the fallback takes over, the tone changes. ORA now keeps the same model for the life of a conversation when it can, and only switches at the start of a new turn.

When you should not do this

If your product is one model and one job, a router is overhead. Call the API. The routing layer earned its place in ORA because the product promise is "the right model for each thing", and because I ship to four platforms where app updates are slow.

If you are building something similar, the order I would do it in now: define your task list first, make the client send tasks and nothing else, put a registry behind one endpoint, and add the fallback chain before any of the clever scoring. The clever scoring can wait. The fallback cannot.

ORA is on the App Store and Google Play. If you have built a router that does something smarter than this, I would like to hear how it chooses.