The router from Part 4 is a model call, and model calls change when you edit a prompt, swap a model or add a tool group. This part adds a 21-case eval that checks the router in about half a minute and exits non-zero when it gets worse. It is small on purpose, and I'll say plainly what it can't tell you.
Prerequisites: Part 3 and Part 4.
Why the router needs a test
Every request in this agent starts with one model call that answers a single word: filesystem, geoip, webcrawler or general. If that word is wrong, everything after it is wrong in a quiet way. The filesystem agent never sees the IP tools, so a misrouted "Where is 1.1.1.1 located?" gets answered by an agent that has no way to look it up. Nothing crashes. You just get a confident, useless answer.
When I tuned the router prompt by hand, I fixed one prompt and had no idea whether I'd broken another. I was re-typing the same dozen questions into the terminal and reading the ⇢ route: line. An eval is that habit written down.
Make the router importable
The router started as a function nested inside build_graph, which is a fine place for a graph node and a bad place for a test to reach. The first change was to pull the model call out into a top-level function that both the graph node and the eval can use.
agent_graph.py · classify
async def classify(model: ChatOpenAI, message: BaseMessage) -> str:
"""Pick a route for one user message. Falls back to 'general' on unparseable output."""
reply = await model.ainvoke([SystemMessage(ROUTER_PROMPT), message])
text = str(reply.content).lower()
return next((r for r in TOOL_GROUPS if r in text), "general")
It sends the router prompt plus one message, lowercases the reply and returns the first route name that appears in it. If the model rambles or answers something unexpected, the fallback is general. The router node is now one line that calls it:
agent_graph.py · router (inside build_graph)
async def router(state: State) -> dict:
"""Node: route on the newest user message only."""
return {"route": await classify(model, state["messages"][-1])}
The point of the refactor is that the eval doesn't test a copy of the router. It imports classify and calls the same code the live graph calls. If I change ROUTER_PROMPT, both change together.
The eval
The whole eval is one file, eval_routing.py. The data is a list of (prompt, expected route) pairs:
eval_routing.py · CASES
# (prompt, expected route). Add a case whenever the router misroutes something real.
CASES = [
("List the files in ~/projects", "filesystem"),
("How many folders are in ~/Python?", "filesystem"),
("Show me the contents of README.md", "filesystem"),
("Find every file that mentions LM Studio", "filesystem"),
("What is the size of agent.py?", "filesystem"),
("Grep for create_agent in this project", "filesystem"),
("Is 8.8.8.8 a public IP?", "geoip"),
("Where is 1.1.1.1 located?", "geoip"),
("Validate the address 999.1.1.1", "geoip"),
("Show me the GeoIP stats", "geoip"),
("Which hosts in the database are in Germany?", "geoip"),
("Get the title of https://example.com", "webcrawler"),
("Crawl example.com and summarize the pages", "webcrawler"),
("Check if https://example.com is up", "webcrawler"),
("Take a screenshot of example.com", "webcrawler"),
("What does the front page of news.ycombinator.com say?", "webcrawler"),
("What is the capital of France?", "general"),
("Explain what a StateGraph is in one sentence", "general"),
("Thanks, that helps!", "general"),
("What did we just talk about?", "general"),
("Write a haiku about routers", "general"),
]
That is 21 cases: 6 filesystem, 5 geoip, 5 webcrawler and 5 general. The general cases matter as much as the others. A router that sends everything to a tool group would look great on the first three categories and be useless for small talk and follow-ups.
Two of the cases were rewritten before I wrote this up. One used a personal domain and now uses example.com; one used a real local path and now uses ~/projects. If you copy these, replace them with prompts you would really type.
The runner loops over the cases, calls classify, prints a mark per case, then per-route scores and an overall number:
eval_routing.py · main
async def main(threshold: float) -> int:
results = []
for prompt, expected in CASES:
try:
got = await classify(llm, HumanMessage(prompt))
except Exception as e: # most likely LM Studio isn't running
print(f"Could not reach the model: {e}", file=sys.stderr)
return 2
results.append((prompt, expected, got))
print(
f" {'✓' if got == expected else '✗'} {expected:<10} got {got:<10} {prompt}"
)
by_route = defaultdict(lambda: [0, 0]) # expected route -> [correct, total]
for _, expected, got in results:
by_route[expected][0] += got == expected
by_route[expected][1] += 1
print("\nPer route:")
for route, (ok, total) in by_route.items():
print(f" {route:<10} {ok}/{total}")
correct = sum(e == g for _, e, g in results)
accuracy = correct / len(results)
print(
f"\nOverall: {correct}/{len(results)} ({accuracy:.0%}), threshold {threshold:.0%}"
)
return 0 if accuracy >= threshold else 1
The exit code is the useful part. It returns 0 at or above --threshold (default 0.85), 1 below it, and 2 if the model can't be reached, so a script or a pre-commit hook can tell "the router got worse" from "LM Studio isn't running". It makes about 21 short model calls, one per case, and no tool runs, so nothing touches your files or the network.
The result
One run, with LM Studio serving qwen2.5-7b-instruct-mlx. Model output varies, but a single-word classification with this prompt was stable for me:
$ uv run python eval_routing.py
✓ filesystem got filesystem List the files in ~/projects
✓ filesystem got filesystem How many folders are in ~/Python?
✓ filesystem got filesystem Show me the contents of README.md
✓ filesystem got filesystem Find every file that mentions LM Studio
✓ filesystem got filesystem What is the size of agent.py?
✓ filesystem got filesystem Grep for create_agent in this project
✓ geoip got geoip Is 8.8.8.8 a public IP?
✓ geoip got geoip Where is 1.1.1.1 located?
✓ geoip got geoip Validate the address 999.1.1.1
✓ geoip got geoip Show me the GeoIP stats
✓ geoip got geoip Which hosts in the database are in Germany?
✓ webcrawler got webcrawler Get the title of https://example.com
✓ webcrawler got webcrawler Crawl example.com and summarize the pages
✓ webcrawler got webcrawler Check if https://example.com is up
✓ webcrawler got webcrawler Take a screenshot of example.com
✓ webcrawler got webcrawler What does the front page of news.ycombinator.com say?
✓ general got general What is the capital of France?
✓ general got general Explain what a StateGraph is in one sentence
✓ general got general Thanks, that helps!
✓ general got general What did we just talk about?
✓ general got general Write a haiku about routers
Per route:
filesystem 6/6
geoip 5/5
webcrawler 5/5
general 5/5
Overall: 21/21 (100%), threshold 85%
21 of 21. My honest reaction was not "great" but "suspicious". A perfect score on a test I wrote myself, against a prompt I tuned by hand, usually means the test is easy. Which brings up the question of whether this eval can fail at all.
Check that the eval can fail
A test that always passes is worse than no test, because it gives you confidence for free. So I broke the router on purpose, without editing any file: the one-liner below replaces ROUTER_PROMPT in memory with a prompt that always answers "general", then runs the eval.
uv run python -c "import asyncio,sys,agent_graph,eval_routing; agent_graph.ROUTER_PROMPT='Answer with the single word: general'; sys.exit(asyncio.run(eval_routing.main(0.85)))"
This works because classify reads ROUTER_PROMPT from the module each time it is called, so changing the module attribute changes the behavior. Output from one run (the per-case lines are trimmed here):
✗ filesystem got general List the files in ~/projects
✗ filesystem got general How many folders are in ~/Python?
…
✗ webcrawler got general What does the front page of news.ycombinator.com say?
✓ general got general What is the capital of France?
…
Per route:
filesystem 0/6
geoip 0/5
webcrawler 0/5
general 5/5
Overall: 5/21 (24%), threshold 85%
5 of 21, 24%, and the process exits with code 1. Filesystem, geoip and webcrawler all drop to zero while the five general cases still pass, which is exactly what a router that gives up and says "general" should score. The eval can fail, and it fails in the places I'd expect.
Do this once for any eval you write: break the thing under test on purpose and confirm the score drops and the exit code changes. It takes a minute and tells you the harness is wired up.
What a perfect score doesn't tell you
With this 7B model and this prompt, 21/21 is a statement about 21 prompts I wrote. It is not a claim that routing is solved. - The cases are probably easy. Each prompt contains an obvious cue: a file name, an IP address, a URL. Real requests are vaguer ("is that server okay?"). - It tests the route, not the tool. A correct route can still lead to the wrong tool, or to the right tool with a wrong answer. The model miscounting the folders in a directory listing would not show up here at all. - Single messages, no history. The live router sees only the newest user message too, but the follow-up cases ("What did we just talk about?") are tested with no conversation behind them. - Non-deterministic. The same prompt can route differently on another day, another model quantization or another temperature. A pass is evidence, not a proof. - It needs the model running. It is a check against a real model, not a unit test, so it is slower than one and can't run in an environment without LM Studio.
Growing the eval
The comment above CASES says what to do: add a case whenever the router misroutes something real. That is the cheapest way to make the set harder, because each case comes from a failure you actually hit. Two other additions I'd make next:
- Mixed-category prompts, like "Read the file at that URL's saved path" or "Look up the IP in this log file". They are the real stress test for a single-word router, and you have to decide what the expected route is, which is itself a useful design exercise.
- Rephrasings of the existing cases with no obvious keywords, to find out how much the router leans on cues like "file" and "IP".
When a harder case fails, you have a choice: tighten ROUTER_PROMPT, change the route descriptions, or accept the miss. The eval doesn't make that call for you. It makes sure you notice the next time you change something and a different case flips.
Recap
classifywas refactored out of the router node so the eval and the live graph share one piece of code.eval_routing.pyruns 21 cases (6 filesystem, 5 geoip, 5 webcrawler, 5 general), prints per-route and overall scores, and exits 0, 1 or 2.- It scored 21/21 on the current router; a router prompt that always answers "general" scored 5/21 (24%), so the eval can fail.
- It checks only the route, with single messages and easy cases. Add a case whenever a real misroute happens.
Next: Part 6 lets the agent change files, behind a human yes or no.
Python 3.14.0, langgraph 1.2.20, langchain 1.4.10, langchain-openai 1.7.4, qwen2.5-7b-instruct-mlx in LM Studio.