fix(design-skill): tighten predictability checks #18

Merged
alexion merged 1 commits from tighten-design-skill into main 2026-07-30 22:07:06 -04:00
2 changed files with 35 additions and 34 deletions
Showing only changes of commit 43280e4d4a - Show all commits

View File

@@ -1,9 +1,7 @@
# Glossary — Structural Predictability # Glossary — Structural Predictability
The domain model for what makes an Agent Skill structurally predictable. The disclosed domain model for [`design-skill`](SKILL.md).
A skill exists to wrangle determinism out of a stochastic system. Every term below is a lever on **Predictability**.
The root virtue is **Predictability**, and every term below is a lever on it.
This is the disclosed reference for [`design-skill`](SKILL.md).
The terms are grouped by axis: **Invocation** (how a skill is reached), **Information Hierarchy** (how its content is arranged), **Steering** (how the agent's runtime behaviour is shaped), and **Pruning** (how it is kept lean). The terms are grouped by axis: **Invocation** (how a skill is reached), **Information Hierarchy** (how its content is arranged), **Steering** (how the agent's runtime behaviour is shaped), and **Pruning** (how it is kept lean).
Each **failure mode** lives beside the lever that cures it, tagged _failure mode_. Each **failure mode** lives beside the lever that cures it, tagged _failure mode_.
@@ -96,8 +94,9 @@ More **model-invoked** skills spend **context load** as more descriptions crowd
More **user-invoked** skills spend **cognitive load** by giving the human more to remember and reach for. More **user-invoked** skills spend **cognitive load** by giving the human more to remember and reach for.
Two cuts guide the division. Two cuts guide the division.
By **invocation**, split off a model-invoked skill where you have a distinct **leading word** to trigger it — a trigger word you actually use in your prompts. By **invocation**, split off a model-invoked skill where you have a distinct **leading word** to trigger it — a trigger word you actually use in your prompts.
By **sequence**, split a run of **steps** where a step's **post-completion steps** need hiding, since isolating it in its own context clears what follows. By **sequence**, split a run of **steps** only when the hand-off creates a real context boundary that hides a step's **post-completion steps**.
Beware the reverse: merging sequences exposes each step's post-completion steps to what follows, inviting premature completion. Beware the reverse.
Merging sequences exposes each step's post-completion steps to what follows, inviting premature completion.
_Avoid_: chunking, modularity _Avoid_: chunking, modularity
@@ -200,8 +199,9 @@ Reach for an existing word first.
A leading word serves **predictability** twice. A leading word serves **predictability** twice.
In the body it anchors **execution** — the agent reaches for the same behaviour every time the concept appears, and inside flat reference it focuses attention on a class of thing to look for, recruiting the right checks each run. In the body it anchors **execution** — the agent reaches for the same behaviour every time the concept appears, and inside flat reference it focuses attention on a class of thing to look for, recruiting the right checks each run.
In the **description** it anchors **invocation** — and not only within the skill: when the same word lives in your prompts, your docs, and your codebase, the agent links that shared language to the skill and fires it more reliably. For a **model-invoked** skill, the **description** anchors **invocation**.
Word a description with the leading words you actually use when you want the skill. When the same word lives in prompts, documentation, and code, the agent links that shared language to the skill and fires it more reliably.
Word a model-invoked skill's description with the leading words actually used to request its behavior.
_Avoid_: keyword, term, motif _Avoid_: keyword, term, motif
@@ -222,7 +222,8 @@ _Avoid_: done condition, exit condition, stopping rule
The work an agent does behind the scenes within a single step — reading files, exploring the codebase, making changes, digging up what it needs rather than offloading to the user. The work an agent does behind the scenes within a single step — reading files, exploring the codebase, making changes, digging up what it needs rather than offloading to the user.
It lives below the step structure: never written as its own step, latent in the wording, controlled by the agent rather than the skill. It lives below the step structure: never written as its own step, latent in the wording, controlled by the agent rather than the skill.
The within-step counterpart to **post-completion steps**' across-step pull. The within-step counterpart to **post-completion steps**' across-step pull.
Raised by a **leading word** (_comprehensive_, _thorough_) or a **completion criterion** that demands the work be exhaustive — including the demand axis applied to flat reference, which is what drives a skill of flat reference to cover all its rungs. Raised by a strong **leading word** or a **completion criterion** that demands exhaustive work.
The demand axis also applies to flat reference and drives a reference-only skill to cover every required check.
Goes thin either when that demand is missing or when **premature completion** cuts the step short. Goes thin either when that demand is missing or when **premature completion** cuts the step short.
_Avoid_: scope, effort, diligence, coverage _Avoid_: scope, effort, diligence, coverage
@@ -272,10 +273,10 @@ _Avoid_: repetition, redundancy
### Relevance ### Relevance
Whether a line still bears on what the skill does — the lens for what to keep. Whether an instruction still bears on what the skill does — the lens for what to keep.
A line loses relevance either by never bearing on the task (mere exposition, or a **branch** that should be disclosed) or by going stale: drifting out of date as the behaviour or world it describes changes. An instruction loses relevance either by never bearing on the task or by going stale as the behavior or world it describes changes.
Shorter skills are easier to keep relevant, because each line is cheaper to check. Shorter skills are easier to keep relevant because each instruction is cheaper to check.
Distinct from **no-op**: relevance asks whether a line bears on the task, not whether it changes behaviour. Distinct from **no-op**: relevance asks whether an instruction bears on the task, not whether it changes behavior.
_Avoid_: load-bearing, staleness, freshness _Avoid_: load-bearing, staleness, freshness
@@ -290,11 +291,11 @@ _Avoid_: accretion, bloat, cruft, rot
### No-Op ### No-Op
_Failure mode._ An instruction that changes nothing because the model already does it by default — you pay load to tell the agent what it would do anyway. _Failure mode._ An instruction that changes nothing because the model already does it by default — you pay load to tell the agent what it would do anyway.
The test: does a line change behaviour versus the default? The test asks whether an instruction changes behavior versus the default.
A line can be perfectly **relevant** and still be a no-op. An instruction can be perfectly **relevant** and still be a no-op.
The same priors that make a **leading word** free make a no-op worthless. The same priors that make a **leading word** free make a no-op worthless.
A leading word is a _technique_, while No-Op is a _verdict_ on a line. A leading word is a _technique_, while No-Op is a _verdict_ on an instruction.
The two concepts cross. The two concepts cross.
A leading word too weak to beat the default is a no-op (_be thorough_ when the agent is already thorough-ish), and the fix is a stronger word that passes the verdict (_relentless_), not a different technique. A leading word too weak to beat the default is a no-op (_be thorough_ when the agent is already thorough-ish), and the fix is a stronger word that passes the verdict (_relentless_), not a different technique.
So the No-Op test — does it change behaviour versus the default? — is also how you grade whether a leading word is earning its repetitions. So the No-Op test — does it change behaviour versus the default? — is also how you grade whether a leading word is earning its repetitions.

View File

@@ -6,12 +6,9 @@ disable-model-invocation: true
# design-skill # design-skill
Draft a new Agent Skill or audit and rewrite an existing one against one bar: **predictability**, meaning that the agent follows the same process on every run. Judge only structural **predictability** using the terms in [`GLOSSARY.md`](GLOSSARY.md).
This skill judges structural predictability only. Do not expand the work into general prose, safety, or technical review.
It does not judge general prose quality, safety, or technical correctness. Read each relevant definition before applying it.
**Bold terms** are defined in [`GLOSSARY.md`](GLOSSARY.md).
Read the relevant definitions before applying them.
## Select the branch ## Select the branch
@@ -71,11 +68,12 @@ Done when the skill conforms to the target project's conventions and every captu
### 1. Locate and read it ### 1. Locate and read it
Read the target project's instructions, locate the named skill, and trace generated or linked placements to their canonical source. Read the target project's instructions and locate the named skill's canonical source.
Ask only when multiple plausible canonical sources remain. Trace generated or linked placements rather than editing them.
Read `SKILL.md`, every file it discloses, and any project documentation it treats as authoritative. Ask only when multiple plausible sources remain.
Read `SKILL.md`, every disclosed file, and any project documentation it treats as authoritative.
Done when the full design and its source-of-truth boundaries are visible. Done when the full design and every source-of-truth boundary are visible.
### 2. Apply the predictability checklist ### 2. Apply the predictability checklist
@@ -91,7 +89,6 @@ Check every relevant definition in `GLOSSARY.md` and quote the exact offending p
Judge sentence by sentence rather than line by line. Judge sentence by sentence rather than line by line.
A line may contain both a load-bearing sentence and a no-op sentence. A line may contain both a load-bearing sentence and a no-op sentence.
Do not expand the audit into general writing, safety, or technical review.
Done when every glossary failure mode has been checked and every finding has quoted evidence. Done when every glossary failure mode has been checked and every finding has quoted evidence.
@@ -109,18 +106,21 @@ Done when every finding is fixed or explicitly retained with a structural reason
Run the target project's required checks and any available Agent Skills validator. Run the target project's required checks and any available Agent Skills validator.
At minimum, verify frontmatter, naming, relative references, disclosed-file existence, and invocation compatibility with the target harness. At minimum, verify frontmatter, naming, relative references, disclosed-file existence, and invocation compatibility with the target harness.
Do not claim a check ran when its tooling is unavailable.
Done when every available deterministic check passes or a concrete limitation is reported. Done when every available deterministic check passes and every unavailable check is reported.
### 2. Smoke-test behavior ### 2. Smoke-test behavior
Derive one realistic prompt from the request and the skill's intended trigger phrasing. Derive one realistic task prompt from the skill's intended trigger phrasing.
When the harness supports isolated subagents or equivalent clean runs, run the prompt once with the skill and inspect whether the agent followed the intended branch, hesitated, misread an instruction, or behaved unexpectedly. Launch a fresh context through an actual subagent tool or documented headless harness command, load the skill there, and run the prompt.
The fresh context must attempt the target skill's task rather than design or audit that skill.
Save the exact prompt and fresh-context output in temporary files outside the target project.
Inspect that output for whether the agent followed the intended branch, hesitated, misread an instruction, or behaved unexpectedly.
Do not count the current design run, a self-review, or deterministic validation as behavioral evidence.
A smoke test does not require advance approval. A smoke test does not require advance approval.
If no isolated execution mechanism exists, report that limitation rather than treating self-review as behavioral evidence.
Done when the smoke test succeeds without structural confusion or its unavailable mechanism is reported. Done when the final report names the fresh-context mechanism and both evidence paths, and the captured output shows no structural confusion.
If no isolated execution mechanism exists, report that limitation instead.
### 3. Offer stronger verification when available ### 3. Offer stronger verification when available
@@ -141,4 +141,4 @@ Follow explicit user or project instructions for staging or other version-contro
Otherwise leave changes unstaged and report every changed path. Otherwise leave changes unstaged and report every changed path.
Never commit unless explicitly instructed. Never commit unless explicitly instructed.
Done when no structural finding remains, required checks pass, behavioral confidence is stated accurately, and changed paths are reported. Done when no structural finding remains, required checks pass, behavioral confidence is stated accurately, and the applicable version-control policy was followed.