AI Agent Tool Schemas Cut 47%: The Hidden Trap of Duplicate YAML Keys

When a self-hosted AI Agent responds slowly, most people's first instinct is to swap the model or add more VRAM. But the real bottleneck is often hidden somewhere more subtle — on every request, the Agent keeps re-describing tools that were never actually called. A developer running a self-built Hermes Agent system cut the fixed prompt by 27% by trimming tool schemas, and along the way hit a duplicate YAML key issue that made the change look like it worked while actually doing nothing at all.

Where the time actually goes

The developer runs Hermes Agent with several specialized profiles — coding, review, scouting, security, and writing — plus a default "chief of staff" profile, all pointing at a local model on a 6GB GPU. A single conversational turn often exceeded a minute and a half, which by intuition should be blamed on insufficient model capability.

Testing ruled out that hypothesis: larger candidate models wouldn't fit in 6GB of VRAM at any context length, and the closest alternative tied on key metrics. What really mattered was the prefill phase — the process of handling the prompt before the model returns its first token. A real request capture showed ~18,400 tokens of prompt, of which 13,541 (74%) were the JSON schemas of 27 enabled tools. Measured another way, tool schemas accounted for 58%–67% of each profile's fixed prompt. At roughly 600 tokens/sec prefill speed on that machine, a large chunk of every turn was spent before the model even "started thinking."

The conclusion was clear: the leverage wasn't in the model, it was in the tool list.

The division of labor between prefill and decode. LLM inference is typically split into two phases: prefill and decode. Prefill is where the model processes the entire input prompt in parallel and builds the KV Cache — it determines the time to first token (TTFT), and its duration can be approximated as prompt tokens ÷ prefill throughput (tokens/sec). Decode is where the model generates output token by token, constrained by memory bandwidth and model size. In self-hosted, multi-turn scenarios, if the system prompt and tool schemas are long while outputs are short, prefill often dominates the per-turn latency — which is exactly the main cause of the 90+ second single-turn times on the 6GB GPU described here. That's also why cutting the tool schemas, which take up the largest share of the prompt, speeds things up more immediately than swapping in a bigger model.

KV Cache. The key-value cache (KV Cache) built during prefill stores the attention intermediates for already-processed tokens, so that subsequent decode steps don't need to recompute the prefix. It directly consumes VRAM, so the longer the prompt, the less VRAM is left for generation length and concurrency. On memory-constrained machines, trimming the fixed prompt not only saves time but also indirectly increases the available context budget.

Measure first, then act

Hermes provides an offline command, hermes prompt-size --json, that breaks down the composition of the fixed prompt: system prompt, skill index, memory, tool count and total schema bytes, plus a per-toolset breakdown.

hermes --profile coder prompt-size --json

One easy pitfall: the per-toolset breakdown array is called toolsets_breakdown, not toolsets.

To get real token counts rather than estimates, the developer sent text to Ollama's /api/generate endpoint and read the prompt_eval_count from the response. For these schemas, the measured ratio was about 3.97 bytes/token, which lets the byte counts from prompt-size be converted conveniently.

They also used a packet-capture rig previously built for debugging other issues: a service masquerading as an OpenAI-compatible endpoint that records what actually gets transmitted. Two lessons: keep the largest request in a run rather than the first (the initial call usually doesn't include tools); and label requests from files rather than reading from server-side environment.

Why byte counts need to be converted to tokens. hermes prompt-size outputs schema byte counts, but model billing and inference cost are measured in tokens. The byte-to-token ratio varies widely across tokenizers and character sets: English text is roughly 4 bytes/token, Chinese is about 1.5–3 bytes/token, and JSON schemas — due to heavy punctuation, key names, and repeated structures — commonly land between 3.5 and 4.5. The measured ~3.97 bytes/token here is a typical value. Using a count returned directly by the inference engine, like prompt_eval_count, is more reliable than dividing character counts by an empirical coefficient, and it avoids misjudging the optimization's effect. If you don't have a real count handy, you can cross-calibrate using the usage field returned by Ollama's /api/generate or an OpenAI-compatible endpoint.

Decide based on usage records

Rather than trimming tools on intuition, the developer first checked what the Agent actually called. Hermes stores each profile's message history in a local SQLite database, so they tallied tool calls across roughly 1,900 messages.

• Ten browser_* tools, text_to_speech, clarify, and process had zero calls.
• session_search and delegate_task were used, but only on the default profile (5 and 21 times respectively).

Based on that, they made evidence-backed cuts:

profile	disabled toolsets
default	tts
coder, reviewer, scout, sec, writer	browser, tts, clarify, session_search, delegation

One thing had to be verified before rollout: the browser toolset is a bundle of 13 tools that includes web_search, which the Agent actually uses. On the packet-capture rig, disabling browser removed only the 10 browser_* tools, while web_search survived because it's a protected core tool. Judging by the bundle list alone would have led to the wrong conclusion; only empirical testing confirmed it.

Trap 1: Duplicate YAML keys

The config item is agent.disabled_toolsets in each profile's config file. The developer added it right after the agent: line:

agent:
  disabled_toolsets: [browser, tts, clarify, session_search, delegation]
  # ... lots of other config ...
  disabled_toolsets: []

It looked fine. grep -A6 showed the new list right where expected. But it did absolutely nothing.

Further down in the same agent: block, the file already had disabled_toolsets: []. YAML doesn't error on duplicate keys in a mapping; PyYAML simply keeps the last one. So the new list was parsed as an empty list, all tools stayed enabled, and there was no warning of any kind.

The fix is mundane but important:

1. Count keys before adding: run grep -c disabled_toolsets on the file. If it already exists, edit the existing line.
2. Verify with real parsing, never with grep:

python3 -c "import yaml; print(yaml.safe_load(open('config.yaml'))['agent']['disabled_toolsets'])"

grep shows text; only a parser shows what the program actually sees. Later, when writing a helper script to edit profile configs, the developer used a YAML loader that rejects duplicate keys, backed up first, re-read to verify after writing, and errored out on failure.

Why duplicate keys get silently swallowed. The YAML spec requires keys in a mapping to be unique, but it doesn't force parsers to error on duplicates — handling is left to the implementation. The default strategy of mainstream implementations like PyYAML is "last one wins": the last occurrence is kept, with no warning whatsoever. This creates a dangerous contrast with other config formats: JSON parsers also typically take the last value for duplicate keys, while many static-check tools (like certain yamllint rules) only check indentation and syntax, not semantic duplication. So two disabled_toolsets entries in one file can legally "pass validation" while silently invalidating the earlier config. There are three defenses: first, count key occurrences with grep -c before editing; second, use a loader like ruamel.yaml that errors on duplicate keys or preserves comments; third, immediately assert the written value by reading it back with yaml.safe_load, separating "textually correct" from "semantically correct" verification.

Trap 2: CLI can't write lists

The alternative to manual editing is the command line:

hermes config set agent.disabled_toolsets '["browser","tts"]'

Equally ineffective. It wrote the value as a quoted string:

disabled_toolsets: '["browser","tts"]'

The filter expects a list, silently ignores the string, and tts is still there when you check. List values must be written as real YAML sequences by editing the file.

Both traps have the same shape: the config appears set, no tool errors out, and behavior doesn't change. Only measurement can reveal the problem.

The mismatch between CLI and YAML's type system. A command line can only pass strings, whereas YAML distinguishes scalars, sequences, and mappings as three composite structures. In hermes config set agent.disabled_toolsets '[...]', the bracketed argument is passed through verbatim by the shell, and the config tool writes it as a plain string — producing disabled_toolsets: '["browser","tts"]' in the YAML, which is syntactically a valid scalar string, not a list. If downstream code uses if "browser" in disabled_toolsets, the string-containment check might even "accidentally" match, but in most implementations using set() or list equality comparison, it will be ignored. Such config tools should support explicit type markers (like --type json) or assert isinstance(value, list) after writing; if the tool doesn't support it, any list- or dict-typed config should go back to file editing with parser-based type verification.

The results of trimming

A before-and-after comparison from prompt-size:

profile	tool count	tool schema bytes	total fixed prompt
specialized profiles (×5)	27 → 13	46,403 → 24,795 (−47%)	80,057 → 58,262 (−27%)
default	24 → 23	50,058 → 48,188 (−3.7%)	74,699 → 72,829

Each specialized profile saves roughly 5,400 tokens per turn, equivalent to about 9 seconds at the measured prefill speed.

After the change, they verified the places that could break: the gateway started and ran normally, the coding Agent correctly executed real terminal commands, and it could still reach the MCP server — the tool_search/tool_call/tool_describe trio in front of it survived the cuts. The default profile kept web search and delegation. All files were backed up beforehand, so rollback just meant restoring files and restarting the gateway.

What's left

The default profile has 5 memory integration tools totaling 7,663 bytes that were never called. They don't appear under regular toolsets because the memory plugin injects them directly, out of reach of disabled_toolsets. Removing them would mean disabling that integration, which is a product decision rather than a simple trim.

The remaining large toolsets — file, skills, terminal, memory — are all in active use. The other half of the fixed prompt is now the system prompt (~21–24 KB) and the skill index (~9 KB), which is the next direction worth watching.

Key takeaways

1. Measure the prompt before blaming the model. On small-VRAM GPUs, prefill dominates, and tool schemas can make up most of it.
2. Trim based on recorded usage. Message history is the evidence; zero calls across ~1,900 messages is a clear signal.
3. Never trust a config edit verified with grep. Use a parser. Duplicate YAML keys don't error; the last one wins.
4. Check how your CLI writes lists. A list quoted into a string looks fine and is actually useless.
5. Re-measure after every change. The only proof a trim worked is that the byte count went down.