Member-only story
LLM Routing with Preference Data
Not every prompt deserves your most expensive model
Asking an LLM to summarize a short email and asking it to reason through a difficult math problem are very different workloads. Yet many production systems treat them the same way: send everything to the strongest available model and accept the bill.
RouteLLM asks a more interesting question: can we learn, from previous model comparisons, which prompts actually need the expensive model?
The answer from the paper is yes, sometimes by quite a lot. The more important finding, to me, is that the quality of the routing data matters at least as much as the router itself.
What the paper is about
RouteLLM, by Isaac Ong and colleagues, tackles the cost versus quality trade-off in LLM serving.
The setup is simple. You have a strong model, such as GPT-4 (for 2024 when the first version of the paper came out), that tends to give better answers but costs more, and a weak model, such as Mixtral-8x7B, that is much cheaper but less reliable. A router looks at each incoming prompt and chooses which model should handle it.
Think of a customer-support application where the input is
“Rewrite this response to…
