Avoiding Pitfalls in Agent Skill Maintenance: A Practical Guide to Defending Against Perplexity's "Action at a Distance" and Building Evaluation Systems

Introduction: Fixed A, Broke C

Have you ever encountered this situation — you modify a Skill's description, or just add a new Skill, and suddenly another Skill you never touched starts misfiring?

Perplexity uses a physics term to describe this risk: Action at a Distance. In physics, it refers to one object instantaneously affecting another without any contact between them. This concept has centuries of debate behind it — Newton's theory of universal gravitation first raised the issue: two celestial bodies with no visible medium connecting them, yet gravity transmits instantaneously. Newton himself was deeply troubled by this, and Einstein later used "spooky action at a distance" to criticize quantum entanglement. In the context of Agent development, this analogy is remarkably precise: you change one Skill's configuration, and another seemingly unrelated Skill breaks.

In an Agent Skill system, this isn't mysticism. When a large language model makes routing decisions, it looks at all Skill names and descriptions together. All Skill Descriptions are concatenated into a single context window fed to the model, and the "medium" between them is the model's Self-Attention mechanism. When the model processes routing decisions, the attention mechanism establishes association weights across all Skill descriptions. A newly added keyword can change how attention is distributed, much like how measuring one particle in quantum entanglement instantaneously affects the state of another particle. Adding a new Skill, or slightly modifying an existing Skill's description, essentially redistributes the model's global attention. A single new keyword can shift routing boundaries, stealing tasks that should go to Skill A, or interfering with the model's judgment of existing skills.

What were once clear business boundaries — well water not mixing with river water — suddenly fall into chaos because of one sentence you added.

Why Brute-Force Fixes Don't Work

Many teams maintain Agent Skills in a very crude way: when something fails, they modify the description or the main instructions; next time it breaks, they modify it again. In the short term, the problem appears fixed, but long-term this approach is highly likely to trigger action at a distance, causing the entire Skill routing system to break down.

Perplexity's core philosophy is: Don't make sweeping changes to the main instructions based on gut feeling. Every failure must first be diagnosed by type, different failures require different fixes, and all modifications must be validated through evaluation to confirm nothing else was broken.

Three Types of Failures, Three Types of Fixes

Type 1: Routing Issues — A Skill loads when it shouldn't, or doesn't load when it should

This problem lies in the Description. A Description is essentially a routing trigger that determines when the model loads this Skill.

To understand the "routing space" here, you need to recognize that what the LLM is doing is essentially a multi-class classification task: given user input, select the most matching one or more Skills from N options. The LLM doesn't route using if-else rules like traditional software — it performs "soft matching" through semantic similarity and contextual reasoning. This means boundaries between Skills aren't hard-coded but determined by the distribution of Description semantic vectors in high-dimensional space. When two Skills' Descriptions are too close in semantic space, the model easily confuses them. This also explains why the traditional software engineering intuition of "module independence" breaks down in Agent development — you think two Skills are completely unrelated, but in the model's semantic space they might be neighbors.

All Skill Descriptions share the same routing space — changing one word can steal another Skill's traffic.

Perplexity's rule is clear: Modifying a Description must be paired with an Eval (evaluation).

• Loads when it shouldn't: Tighten the description, and immediately add negative evaluations to prove you haven't stolen traffic from other Skills.
• Doesn't load when it should: Add keywords, and immediately add positive evaluations to prove it can be correctly triggered.

Modifying a Description without pairing it with an Eval is planting a landmine in your Agent system.

Type 2: Execution Issues — Routing is correct, but execution results are wrong

The Skill loads correctly, but the output is wrong. In this case, Perplexity proposes a very restrained Prompt engineering maintenance principle: Append Mostly.

This strategy reflects a paradigm shift in Prompt engineering from "one-time design" to "progressive evolution." Early Prompt engineering favored pre-designing a perfect instruction set, but practice has proven this approach unsustainable in complex systems. The core insight of the append strategy is: LLMs exhibit nonlinear sensitivity to instructions — minor changes to main instructions can trigger global behavioral drift, while appending specific boundary conditions at the end is relatively safe. This aligns closely with the "Open-Closed Principle" in software engineering (open for extension, closed for modification).

What you append isn't large blocks of explanation stuffed into the main instructions, but rather Gotchas — pitfalls left behind by real failures. Those edge cases that seem like they should obviously work one way, but actually shouldn't. Gotchas are essentially "exception boundary declarations," similar to edge case handling in programming, but existing in natural language form within the Prompt system. For example:

When encountering error X, do not retry — raise an alert immediately.

A minimal negative edge case, appended to a dedicated attachment file, without polluting the main instructions.

This is Perplexity's Gotcha Flywheel mechanism: every failure becomes a boundary, and Skills don't get stronger through rewrites — they get more stable through accumulated boundaries. Going from 80% success rate to 99.9% isn't achieved by making Instructions thicker, but by tightening failure boundaries.

Type 3: Global Prompt Changes — The most insidious form of action at a distance

This situation is harder to detect than the previous two: your Skill hasn't changed a single line, the Description is fine, routing is correct, but the System Prompt — the thing that defines the Agent's underlying behavioral logic — has changed.

Maybe another team changed it, and they had no idea your Skill existed. Certain instructions in your Skill now logically conflict with the new System Prompt. For example, your Skill originally assumed the model would "confirm before executing," but the new Prompt requires "execute by default, report afterward." Your Skill hasn't changed, but the foundation beneath it is shifting.

This situation is particularly insidious because different LLMs handle priority conflicts between System Prompts and Skill instructions differently. Some models strictly follow System Prompt settings, prioritizing the System Prompt when it conflicts with Skill instructions; others are more easily overridden by later-appearing Skill instructions. This uncertainty makes it nearly impossible to predict the impact scope of global Prompt changes through static analysis.

Perplexity's approach is: Investigate immediately — check whether the current Skill has logical conflicts or content duplication with the new settings. This is a reactive defense measure, but one that must be properly executed in LLM engineering practice.

Four-Layer Evaluation System: The Safety Net for Modifications

Whether you're tightening a Description, adding keywords, or appending Gotchas, they all raise the same question: how do you know this modification hasn't broken something else? You can't rely on gut feeling.

Perplexity's answer is to build a multi-layer evaluation suite — and not just testing the final result, but decomposing Skill failures into multiple layers for testing:

Layer 1: Routing Tests

Does it load when it should? Does it mistakenly load when it shouldn't? First ensure Skill routing is correct — this is the foundation of all evaluation.

Layer 2: Path Tests

Just because a Skill is loaded doesn't mean the model actually reads the content it should. If important content is in References or Assets, the model still needs to expand them under the right conditions. This layer tests: Does it read attachments when it should? Does it read them when it shouldn't? After reading, does it use them correctly?

Layer 3: End-to-End Tests

Run the Agent's complete cycle, then use an LLM as a judge (LLM-as-Judge) to score the task's final result against clear evaluation criteria.

LLM-as-Judge is an important methodology that has emerged in AI evaluation over the past two years. Traditional automated testing relies on exact matching or regular expressions, but Agent outputs are often open-ended natural language that can't be judged right or wrong with hard-coded rules. The LLM-as-Judge approach provides the judge model with scoring criteria (Rubric), reference answers, and the output to be evaluated, having it score along predefined dimensions. Note that this method has its limitations — research shows LLMs tend to give higher scores to longer responses (verbosity bias) and to options appearing first (position bias). Therefore, scoring criteria should be as specific and quantifiable as possible, avoiding vague subjective judgments.

Layer 4: Cross-Model Tests

This is a layer many Agent developers overlook. GPT and Claude/Sonic have vastly different behavioral logic when processing skills — a boundary you appended that works on one model might completely fail on another. Testing must cover all underlying LLMs you use.

Different LLMs exhibit significant behavioral differences when processing the same Prompt, stemming from multiple technical levels. First, differences in training data and RLHF (Reinforcement Learning from Human Feedback) preferences: GPT series tends to more proactively execute instructions, while the Claude series leans toward cautious confirmation and refusal. Second, different context window processing strategies — some models are more sensitive to content appearing earlier in long contexts (primacy effect), others are more sensitive to content appearing later (recency effect), directly affecting the probability of Gotchas appended at different positions taking effect. Additionally, different models weight System Prompts differently — some strictly follow the System Prompt, while others are more easily overridden by User Messages. These differences make cross-model testing not a "nice-to-have" but a hard requirement in production environments.

Perplexity's Three Pieces of Advice for Developers

At the end of their handbook, Perplexity leaves all Agent developers with several core recommendations:

1. Use Skills to Automate Your Own Daily Work

If you don't use Agents to automate your weekly routine tasks — like having an Agent do code reviews or write incident postmortems — you'll struggle to write good Skills. The more you see, the more you use, the better you'll understand where the boundaries are.

2. Evaluation Must Come First

Before writing a new Skill, write the Eval first, especially negative tests targeting adjacent but different skills.

This philosophy is directly aligned with TDD (Test-Driven Development) in software engineering, but carries unique significance in AI engineering. In traditional TDD, test cases define the expected behavior of code; in Agent development, Evals not only define a Skill's expected output but more importantly define what a Skill "should NOT do" — i.e., negative tests. This is because the behavioral space of LLMs is far larger than deterministic code, and a Skill might be triggered on inputs you never anticipated. The essence of negative testing is drawing "forbidden zones" for Skills in the routing space to prevent them from crossing boundaries. This mindset requires developers to think about a Skill's boundaries with adjacent Skills during design, rather than patching things up after production incidents.

3. Less Is More

Never start by writing a ten-thousand-word essay for the model. A Skill should begin as a thin sheet of paper — don't try to anticipate how the model might fail. Let it run in real environments, wait for it to actually fail, then let it naturally grow in depth through the Gotcha flywheel.

Conclusion

A good Agent Skill isn't written in one brilliant stroke of genius — it's incrementally appended under the protection of an evaluation system, through real failures, one piece at a time. If you just modify descriptions and add rules based on gut feeling after each failure, your Skill system will inevitably spiral out of control.

Only by compressing every failure into a precise Gotcha, iterating carefully under the protection of an evaluation suite, can Skills truly transform from a pile of instructions into stable, reliable system assets. Remember: maintaining Skills isn't writing an essay — it's performing surgery. Every cut must be precise, and every cut must be verified.