Case Study — Perplexity · Miraptor Docs

Source: Perplexity’s internal essay Designing, refining, and maintaining agent skills at Perplexity, based on operational experience with their agent product Computer.
Core thesis: Writing a skill is not writing software; it is constructing context for the model and its execution environment.
This page: Walks through the article section by section in its original order, with diagrams at three key concepts. Our own distillation is reserved for the final section.

Opening — skill writing is context engineering, not software engineering

The article’s foundational claim, made in the opening: writing a skill is not writing ordinary software. It is building context for the model and its execution environment. Skills carry their own constraints and their own design principles, and most code-review reflexes do not transfer.

To make that operational, the article inverts PEP 20 side-by-side:

PEP 20	Skill engineering version
Simple is better than complex	Complexity is the feature
Explicit is better than implicit	Activation is implicit pattern matching
Sparse is better than dense	Context is expensive; maximum signal per token
Special cases aren’t special enough	Gotchas ARE the special cases
If it’s easy to explain, it’s a good idea	If it’s easy to explain, the model already knows it; delete it

The observation that follows: engineers reflexively write skills as if they were internal documentation, and review effort goes mostly into cutting them down.

What is a skill?

The article decomposes “skill” into four orthogonal properties.

A skill is a directory

Not a single file. The canonical layout:

SKILL.md — frontmatter + instructions
scripts/ — executable code the agent calls instead of reconstructing
references/ — heavyweight conditional documentation
assets/ — templates, schemas, data
config.json — first-run user setup (e.g. mapping team-specific identifiers like Slack channels)

The directory’s internal organization is itself a skill-engineering problem. The article’s case in point:

Perplexity built a skill covering all 1,945 sections of the U.S. Internal Revenue Code. The first version dropped every section into a single folder. The reported outcome: performance worse than the no-skill baseline. The fix was a three-level hierarchy — about 20 broad categories, about 15 topic clusters per category, then specific section content — paired with custom search utilities and quick-reference guides. Routing reliability improved at each level. For dense reference material, the index into the material is itself the skill-engineering problem.

A skill is a format

Frontmatter requires name (lowercase, hyphens allowed) and description. The article’s framing: the description is a routing trigger, not documentation. Phrasing should be Load when … rather than This skill does …. Optional fields include depends: for hierarchical dependencies and a metadata: block. Whether frontmatter is stripped before the model sees the body is implementation-defined; Computer strips it.

A skill is invocable

Computer’s flow: the agent calls load_skill(name="...") → the skill directory is copied into an isolated sandbox → entries under depends: are auto-loaded recursively → frontmatter is stripped before the model reads the body.

Invocation crosses three cost tiers:

Index — about 100 tokens per skill. Name + description for every non-hidden skill is injected into the system prompt at conversation start. Paid every session, every user, always.
Load — about 5,000 tokens. The full SKILL.md body. Loaded on load_skill and sticky until the next compaction boundary. Multiple skills accumulate; the reported steady state is 3–5 skills per thread.
Runtime — unbounded. scripts/, special-case files, sub-skills, formatting guides. Paid only when the agent actively opens a specific file.

This stratification is the physical basis for every design tradeoff downstream: description bytes are paid on every call, body bytes are paid for the whole post-load thread, and only runtime content is genuinely on-demand.

A skill is progressive

Expensive content loads are deferred to the moment they are needed. This is what keeps the three-tier model economically viable across a catalog of dozens of skills.

When do you need a skill?

When a skill is necessary

The article’s criteria: the task requires behavior change beyond a single-prompt instruction; the agent will fail or perform inconsistently without specialized context; knowledge is durable but absent from training data — cutoffs, enterprise workflows, taste and judgment.

The example given: design skills written by Perplexity’s Henry Modisett, specifying font choices, visual aesthetics, and design principles. Judgment-based knowledge the model cannot infer from training data — a canonical “must be a skill” case.

When a skill is unnecessary

Three cases:

The model already knows it. Standard command sequences are the canonical example. The contrast given:
- Bad: writing out a literal git log; git checkout main; git checkout -b <clean>; git cherry-pick <commit> sequence
- Good: “Cherry-pick the commit onto a clean branch. Resolve conflicts preserving intent.”
It is universal. Universal instructions belong in the global system prompt, not in a conditional skill.
It changes faster than you can maintain it. Rapidly-shifting remote endpoints are the example. A stale skill is worse than no skill: it routes confidently, then misleads.

Critical principle — every skill is a tax

The article’s central discipline. For any sentence in a skill, ask whether the agent would get the task wrong without it. If not, the sentence does not belong.

Brevity is framed as labor-intensive, with a nod to Pascal’s old remark that a short letter takes longer to write than a long one.

A separate finding the article cites: LLM-authored skills, on average, provide no benefit — models can consume procedural knowledge reliably but cannot reliably author it.

How to build a skill

The article gives six steps.

Step 0 — Write evals first. Source them from real production queries, known failures (the case that motivated the skill), and boundary-adjacent queries that should route to a different skill. Negative cases carry more weight than positive ones.

Step 1 — The description (the hardest step). Get the routing trigger right before writing any workflow. The article’s four-item checklist:

Start with Load when …
Target 50 words or fewer
Describe user intent in language drawn from 2–3 real production queries
Do not summarize the workflow; that belongs in the body

The illustration given: a PR Monitoring skill whose description uses real user language — “babysit”, “watch CI”, “make sure this lands” — rather than a functional summary like “monitor pull request status”. The former matches how users actually phrase the request; the latter only matches how an author imagines users phrase it.

Step 2 — Write the body. Restating the opening principle: communicating a workflow to an LLM is categorically different from communicating it to a colleague or even to a runtime system. Skip what the model already knows; avoid prescriptive command sequences; concentrate on gotchas and negative examples; push conditional or heavyweight content into accessory files.

Step 3 — Use hierarchy. Distribute material across scripts/ (“give it code to compose, not reconstruct”), references/ (triggered loads such as “read api-errors.md if the API returns non-200”), assets/ (output templates), and config.json (first-run setup).

Step 4 — Iterate on a branch. Small wording changes in the description produce outsized routing shifts. Submit a single changeset with a complete eval set rather than absorbing incremental review cycles.

Step 5 — Ship.

How to maintain a skill

The gotchas flywheel

Long-term, gotchas are the highest-value content in a skill. Maintenance is append-mostly:

Agent fails in production → append a gotcha
Wrong skill loads → tighten the description, add a negative eval
Skill fails to load when it should → add keywords, add a positive eval
System prompt changes → audit for duplication and contention

Steady-state SKILL.md grows slowly and asymmetrically; what accumulates is negative knowledge.

Eval suite — four categories

Perplexity reports running:

Skill loading precision and recall, including forbidden-load checks
Progressive-loading evals — when the body instructs the agent to read an accessory file such as FORMATTING.md, does it actually read it?
End-to-end domain completion graded by an LLM judge against a rubric
Multi-model evals against GPT, Claude Opus, and Claude Sonnet. The reported observation: Sonnet and GPT behave quite differently when it comes to skills.

Action at a distance

Adding a new skill can degrade existing ones, because the new description competes for the same routing attention. This is the strongest argument for keeping evals catalog-wide rather than skill-local — the regression that matters is whether adding skill X causes skill Y to mis-route.

Our distillation

Six points worth carrying out of the reading. These are our distillation, not the article’s wording:

Writing a skill is context engineering for the model and its execution environment, not software engineering. The PEP 20 inversions, the “Every skill is a tax” discipline, and the prescriptive-command-sequence anti-pattern all follow from this single premise.
A skill catalog is a routing system more than a knowledge base. Cost is paid at the index; signal lives in the description; failures accumulate as gotchas; regressions appear between skills.
The description is the hardest part to write. A single word can change routing behavior.
The gotchas flywheel determines a skill’s long-term value. Steady state is slow accumulation of negative knowledge, not up-front design.
Evals must be written first, must cover negative cases, must run across models, and must be catalog-wide.
Not every plausible candidate should become a skill. Model-known, fast-changing, and universal content stays out.

Source

Designing, refining, and maintaining agent skills at Perplexity — Perplexity Research