The agent from Part 1 runs, but it is only as good as the 7B model behind it. In this part I cover what went wrong when I first used it, the two things I did about it (fewer tools, a stricter system prompt), and the problems that a prompt did not fix.
Prerequisites: Part 1.
Why many tools hurt a small model
Every tool you give an agent is sent to the model on every request as a name, a description and an argument schema. With 39 tools that is a lot of text to read before the model even looks at the question, and many of the names overlap: a file system server can plausibly answer "what's in this folder" with list_dir, tree, find or search_files. A large hosted model usually sorts that out. With qwen2.5-7b-instruct-mlx the wrong-tool picks in this series' title were a real problem, and the project's routing eval exists because of them.
I can't give you a number for how much worse it gets per extra tool; I never measured it that way. What I can say is that cutting the list made the failures I saw easier to reason about, and that Part 4 goes further by showing the model only one group at a time.
Curating the tools
The first lever is simply what you expose. In agent.py the tools are grouped by server, and only the groups listed here reach the model:
agent.py · TOOL_GROUPS, ALLOWED_TOOLS
# Read-oriented subset: fewer tools means fewer wrong picks from a 7B model, and agent.py
# can't write, delete or run commands. Add names here to give it more power.
# Grouped by server; agent_graph.py routes each request to one group.
TOOL_GROUPS = {
"filesystem": {
"list_dir", "tree", "read_file", "head", "tail", "stat", "file_info", "find", "grep",
"search_files", "diff",
},
"geoip": {"validate_ip", "lookup_ip", "query_hosts", "get_stats"},
"webcrawler": {"crawl_website", "list_crawls", "get_crawl_results", "get_page_content"},
} # fmt: skip
ALLOWED_TOOLS = set().union(*TOOL_GROUPS.values())
ALLOWED_TOOLS is the union of the three sets: an allowlist of 19 tool names. select_tools from Part 1 keeps only tools whose name is in it. Anything not listed is invisible to the model, so it can't be called.
What is left out, and why:
- Write tools (
write_file,patch_file,make_dir,touch,copy,move,delete) are collected in a separateWRITE_TOOLSset and kept out of the one-shotagent.py. That script has no approval step, so I did not want it able to change anything. Part 6 adds them back to the graph version behind a confirmation. run_command,run_script,chmodandcreate_symlinkare excluded everywhere. Letting a 7B model run shell commands is not something I wanted to debug.- Tools that need a database. Some tools need a database set up; I left them out of the curated set.
This is a safety choice as much as a performance one. An allowlist is a blunt instrument, but it is the only guard that doesn't depend on the model behaving.
The system prompt, line by line
agent.py · SYSTEM_PROMPT
SYSTEM_PROMPT = f"""You are an assistant with tools for the local file system, IP geolocation, \
and web crawling.
- Use a tool whenever the answer depends on real files, IPs or web pages. Never guess their contents.
- Call one tool at a time and read its result before deciding the next step.
- Use absolute paths. The project root is {MCP_ROOT}.
- Timestamps in tool results are Unix epoch seconds. Never convert them to dates; quote the raw number and say it is epoch seconds.
- If a tool errors, report the error plainly instead of retrying the same call.
- Keep answers short and base them only on tool output."""
Each rule is there because of something the model did, or something I expected it to do:
- The opening sentence tells the model what the tools are for, so it connects "IP" with the geolocation tools rather than answering from memory.
- "Use a tool whenever the answer depends on real files, IPs or web pages. Never guess their contents." Without it a small model will happily describe what a folder "probably" contains.
- "Call one tool at a time and read its result before deciding the next step." This is meant to stop it from guessing the second call's arguments before it has seen the first result. As Part 1 showed, it only sometimes obeys: in my file listing run it requested two tools before reading either result.
- "Use absolute paths. The project root is {MCP_ROOT}." The prompt is an f-string, so the real root is filled in. The model has no working directory of its own, so I give it the root and ask for full paths rather than leave it to guess.
- The epoch rule exists because of the story below.
- "If a tool errors, report the error plainly instead of retrying the same call." Small models like to repeat a failing call. This rule is also my partial answer to the "site is down" problem further down.
- "Keep answers short and base them only on tool output." Shorter answers have fewer places to invent things.
What went wrong
The invented dates
I asked the agent to list the files in a folder and say when it was last modified. With the first setup, with no system prompt at all, the model reported dates like 2022 and 2023. The tool had returned the timestamp 1771544574, which is February 2026.
Symptom: the model gave years such as 2022 and 2023 for a folder whose epoch timestamp, 1771544574, is February 2026. Cause: the file system tool returns Unix epoch seconds, and a 7B model can't reliably turn a ten-digit number into a calendar date. It produces a plausible date instead. First attempt: I added a system prompt rule telling it to convert timestamps "or say it is raw". It still invented dates. Offering conversion as an option invited the model to take it. Fix: I changed the rule to what is in the prompt now: "Never convert them to dates; quote the raw number and say it is epoch seconds." After that the model quoted the number. Lesson: don't ask a small model for date math. Take the task away from it.
| Prompt | What the model did |
|---|---|
| No system prompt | Reported years such as 2022 and 2023 |
| "...convert them or say it is raw" | Still invented dates |
| "Never convert them to dates; quote the raw number and say it is epoch seconds" | Quoted the raw epoch number |
Here is the current prompt on the same kind of question. One run, home path shown as ~, file list trimmed:
$ uv run python agent.py "List the files in ~/langgraph-starter and tell me when the docs folder was last modified"
Using 19 tools
→ list_dir(path='~/langgraph-starter', glob='*.py')
→ stat(path='~/langgraph-starter/docs')
← { "entries": [ { "name": "agent.py", "path": "~/langgraph-starter/agent.py", "type": "file", "size": 3657, "modified": 1794365272.057 … (1033 chars)
← { "path": "~/langgraph-starter/docs", "type": "directory", "size": 96, "created": 1794365327.9784584, "modified": 1794365327.9784584, … (212 chars)
The files in ~/langgraph-starter are:
- agent.py (modified: 1794365272.057856)
- agent_graph.py (modified: 1794360393.7950819)
…
The docs folder was last modified at epoch seconds: 1794365327.9784584.
The raw number is quoted, no date is invented, and the last line says "epoch seconds" as asked. Look closer and you can still see sloppiness: the per-file lines say "modified" without saying epoch seconds, and the glob='*.py' argument is something the model added by itself, so the list of "files" excludes everything that isn't Python. The rule fixed one failure, not the model's habit of narrowing questions.
The miscount
Symptom: I asked how many folders were in a directory that held 5 folders and a .DS_Store file. The model said 6.
Cause: it counted the file as a folder. The tool output marks each entry with type: directory or type: file, but the model counted entries rather than filtering them.
Attempted fix: I added a hint about type: directory versus file. It did not help.
Status: not fixed. I did not find a prompt that cured it, so the project keeps it as a known limit. If the count matters, check the tool output yourself, or have a tool do the counting.
"The website could not be reached"
I asked the agent to check whether a site was up. It called the web crawler's get_page_content tool. The crawler failed with a browser-driver error, and the model told me the website could not be reached.
Symptom: "check if https://example-site is up" produced "the website could not be reached". Cause: two problems. First, the tool failed because of the browser driver, not the website (the real cause was a ChromeDriver and Chrome version mismatch; the full story is in Part 9). The model turned a tool error into a claim about the world. Second, none of the 19 tools returns an HTTP status code, so there is no tool that can truthfully answer "is it up". Fix: partial. The prompt rule "report the error plainly" asks the model to say what the tool said. It narrows the problem but doesn't remove it: whether a small model reports an error plainly or editorialises is still up to the model. The structural answer is a tool that returns a status code, which I did not add. Lesson: a tool list defines what questions the agent can actually answer. When you can't answer a question with the tools given, expect the model to answer anyway.
I did not re-run this one for the series, since the failure came from an environment mismatch rather than from the prompt.
Everything above is what I observed with this 7B model on this machine, one model and temperature 0. It does not mean every small model fails the same way, or that a larger one will not. The prompt changes shifted behaviour on the cases I tried; I did not build a test set for them, and model output varies between runs. The system prompt cannot enforce its rules: "one tool at a time" was broken within a single run above. Treat prompt rules as requests, and the tool allowlist as the enforcement.
What I'd take from this
The prompt can ask for behaviour. It can't guarantee it. Two other mechanisms do the guaranteeing: the allowlist (the model can't call what it can't see) and verification outside the model (check counts and statuses against the tool output). Fewer tools helps, but 19 tools from three unrelated servers are still a lot for one model to read at once. The structural fix is to stop giving it all of them at the same time. A router classifies the request first, then hands a specialised agent only that group's tools. That is Part 4. Before that, Part 3 rebuilds the agent loop by hand, because routing needs somewhere to live.
Recap
- A small model reads every tool description on every request; trim the list to what it needs.
TOOL_GROUPSholds a curated read-only subset;ALLOWED_TOOLSis its union and the only toolsselect_toolslets through. Writes and command execution are excluded from the one-shot agent.- Each line in
SYSTEM_PROMPTanswers a failure I saw or expected. The epoch rule only worked when I removed the option to convert. - Not everything was fixed: the model still miscounts, and it still sometimes turns tool errors into claims about the world.
- Prompts are requests; allowlists and external checks are enforcement.
Next: Part 3