From 43280e4d4ad74a2d46c9bf54a25abafa21510442 Mon Sep 17 00:00:00 2001 From: alexion Date: Thu, 30 Jul 2026 19:34:41 -0400 Subject: [PATCH] fix(design-skill): tighten predictability checks --- skills/design-skill/GLOSSARY.md | 33 +++++++++++++++--------------- skills/design-skill/SKILL.md | 36 ++++++++++++++++----------------- 2 files changed, 35 insertions(+), 34 deletions(-) diff --git a/skills/design-skill/GLOSSARY.md b/skills/design-skill/GLOSSARY.md index b701c5a..649a217 100644 --- a/skills/design-skill/GLOSSARY.md +++ b/skills/design-skill/GLOSSARY.md @@ -1,9 +1,7 @@ # Glossary — Structural Predictability -The domain model for what makes an Agent Skill structurally predictable. -A skill exists to wrangle determinism out of a stochastic system. -The root virtue is **Predictability**, and every term below is a lever on it. -This is the disclosed reference for [`design-skill`](SKILL.md). +The disclosed domain model for [`design-skill`](SKILL.md). +Every term below is a lever on **Predictability**. The terms are grouped by axis: **Invocation** (how a skill is reached), **Information Hierarchy** (how its content is arranged), **Steering** (how the agent's runtime behaviour is shaped), and **Pruning** (how it is kept lean). Each **failure mode** lives beside the lever that cures it, tagged _failure mode_. @@ -96,8 +94,9 @@ More **model-invoked** skills spend **context load** as more descriptions crowd More **user-invoked** skills spend **cognitive load** by giving the human more to remember and reach for. Two cuts guide the division. By **invocation**, split off a model-invoked skill where you have a distinct **leading word** to trigger it — a trigger word you actually use in your prompts. -By **sequence**, split a run of **steps** where a step's **post-completion steps** need hiding, since isolating it in its own context clears what follows. -Beware the reverse: merging sequences exposes each step's post-completion steps to what follows, inviting premature completion. +By **sequence**, split a run of **steps** only when the hand-off creates a real context boundary that hides a step's **post-completion steps**. +Beware the reverse. +Merging sequences exposes each step's post-completion steps to what follows, inviting premature completion. _Avoid_: chunking, modularity @@ -200,8 +199,9 @@ Reach for an existing word first. A leading word serves **predictability** twice. In the body it anchors **execution** — the agent reaches for the same behaviour every time the concept appears, and inside flat reference it focuses attention on a class of thing to look for, recruiting the right checks each run. -In the **description** it anchors **invocation** — and not only within the skill: when the same word lives in your prompts, your docs, and your codebase, the agent links that shared language to the skill and fires it more reliably. -Word a description with the leading words you actually use when you want the skill. +For a **model-invoked** skill, the **description** anchors **invocation**. +When the same word lives in prompts, documentation, and code, the agent links that shared language to the skill and fires it more reliably. +Word a model-invoked skill's description with the leading words actually used to request its behavior. _Avoid_: keyword, term, motif @@ -222,7 +222,8 @@ _Avoid_: done condition, exit condition, stopping rule The work an agent does behind the scenes within a single step — reading files, exploring the codebase, making changes, digging up what it needs rather than offloading to the user. It lives below the step structure: never written as its own step, latent in the wording, controlled by the agent rather than the skill. The within-step counterpart to **post-completion steps**' across-step pull. -Raised by a **leading word** (_comprehensive_, _thorough_) or a **completion criterion** that demands the work be exhaustive — including the demand axis applied to flat reference, which is what drives a skill of flat reference to cover all its rungs. +Raised by a strong **leading word** or a **completion criterion** that demands exhaustive work. +The demand axis also applies to flat reference and drives a reference-only skill to cover every required check. Goes thin either when that demand is missing or when **premature completion** cuts the step short. _Avoid_: scope, effort, diligence, coverage @@ -272,10 +273,10 @@ _Avoid_: repetition, redundancy ### Relevance -Whether a line still bears on what the skill does — the lens for what to keep. -A line loses relevance either by never bearing on the task (mere exposition, or a **branch** that should be disclosed) or by going stale: drifting out of date as the behaviour or world it describes changes. -Shorter skills are easier to keep relevant, because each line is cheaper to check. -Distinct from **no-op**: relevance asks whether a line bears on the task, not whether it changes behaviour. +Whether an instruction still bears on what the skill does — the lens for what to keep. +An instruction loses relevance either by never bearing on the task or by going stale as the behavior or world it describes changes. +Shorter skills are easier to keep relevant because each instruction is cheaper to check. +Distinct from **no-op**: relevance asks whether an instruction bears on the task, not whether it changes behavior. _Avoid_: load-bearing, staleness, freshness @@ -290,11 +291,11 @@ _Avoid_: accretion, bloat, cruft, rot ### No-Op _Failure mode._ An instruction that changes nothing because the model already does it by default — you pay load to tell the agent what it would do anyway. -The test: does a line change behaviour versus the default? -A line can be perfectly **relevant** and still be a no-op. +The test asks whether an instruction changes behavior versus the default. +An instruction can be perfectly **relevant** and still be a no-op. The same priors that make a **leading word** free make a no-op worthless. -A leading word is a _technique_, while No-Op is a _verdict_ on a line. +A leading word is a _technique_, while No-Op is a _verdict_ on an instruction. The two concepts cross. A leading word too weak to beat the default is a no-op (_be thorough_ when the agent is already thorough-ish), and the fix is a stronger word that passes the verdict (_relentless_), not a different technique. So the No-Op test — does it change behaviour versus the default? — is also how you grade whether a leading word is earning its repetitions. diff --git a/skills/design-skill/SKILL.md b/skills/design-skill/SKILL.md index fc5cd02..ca7f312 100644 --- a/skills/design-skill/SKILL.md +++ b/skills/design-skill/SKILL.md @@ -6,12 +6,9 @@ disable-model-invocation: true # design-skill -Draft a new Agent Skill or audit and rewrite an existing one against one bar: **predictability**, meaning that the agent follows the same process on every run. -This skill judges structural predictability only. -It does not judge general prose quality, safety, or technical correctness. - -**Bold terms** are defined in [`GLOSSARY.md`](GLOSSARY.md). -Read the relevant definitions before applying them. +Judge only structural **predictability** using the terms in [`GLOSSARY.md`](GLOSSARY.md). +Do not expand the work into general prose, safety, or technical review. +Read each relevant definition before applying it. ## Select the branch @@ -71,11 +68,12 @@ Done when the skill conforms to the target project's conventions and every captu ### 1. Locate and read it -Read the target project's instructions, locate the named skill, and trace generated or linked placements to their canonical source. -Ask only when multiple plausible canonical sources remain. -Read `SKILL.md`, every file it discloses, and any project documentation it treats as authoritative. +Read the target project's instructions and locate the named skill's canonical source. +Trace generated or linked placements rather than editing them. +Ask only when multiple plausible sources remain. +Read `SKILL.md`, every disclosed file, and any project documentation it treats as authoritative. -Done when the full design and its source-of-truth boundaries are visible. +Done when the full design and every source-of-truth boundary are visible. ### 2. Apply the predictability checklist @@ -91,7 +89,6 @@ Check every relevant definition in `GLOSSARY.md` and quote the exact offending p Judge sentence by sentence rather than line by line. A line may contain both a load-bearing sentence and a no-op sentence. -Do not expand the audit into general writing, safety, or technical review. Done when every glossary failure mode has been checked and every finding has quoted evidence. @@ -109,18 +106,21 @@ Done when every finding is fixed or explicitly retained with a structural reason Run the target project's required checks and any available Agent Skills validator. At minimum, verify frontmatter, naming, relative references, disclosed-file existence, and invocation compatibility with the target harness. -Do not claim a check ran when its tooling is unavailable. -Done when every available deterministic check passes or a concrete limitation is reported. +Done when every available deterministic check passes and every unavailable check is reported. ### 2. Smoke-test behavior -Derive one realistic prompt from the request and the skill's intended trigger phrasing. -When the harness supports isolated subagents or equivalent clean runs, run the prompt once with the skill and inspect whether the agent followed the intended branch, hesitated, misread an instruction, or behaved unexpectedly. +Derive one realistic task prompt from the skill's intended trigger phrasing. +Launch a fresh context through an actual subagent tool or documented headless harness command, load the skill there, and run the prompt. +The fresh context must attempt the target skill's task rather than design or audit that skill. +Save the exact prompt and fresh-context output in temporary files outside the target project. +Inspect that output for whether the agent followed the intended branch, hesitated, misread an instruction, or behaved unexpectedly. +Do not count the current design run, a self-review, or deterministic validation as behavioral evidence. A smoke test does not require advance approval. -If no isolated execution mechanism exists, report that limitation rather than treating self-review as behavioral evidence. -Done when the smoke test succeeds without structural confusion or its unavailable mechanism is reported. +Done when the final report names the fresh-context mechanism and both evidence paths, and the captured output shows no structural confusion. +If no isolated execution mechanism exists, report that limitation instead. ### 3. Offer stronger verification when available @@ -141,4 +141,4 @@ Follow explicit user or project instructions for staging or other version-contro Otherwise leave changes unstaged and report every changed path. Never commit unless explicitly instructed. -Done when no structural finding remains, required checks pass, behavioral confidence is stated accurately, and changed paths are reported. +Done when no structural finding remains, required checks pass, behavioral confidence is stated accurately, and the applicable version-control policy was followed.