feat: add all-skills leaderboard (task 0007) #7

alexion · 2026-07-24T16:58:36-04:00

alexion commented

2026-07-24 16:58:36 -04:00

Task: .claude/tasks/0007-all-skills-leaderboard.md

Summary

Adds the all-skills leaderboard to the benchmark skill's deterministic core.

A second pure transform (build_leaderboard / render_leaderboard) plus a --leaderboard CLI mode that consumes the per-skill *.results.json files and renders tests/.reports/index.html.
Each row carries both verdict badges, links out to its per-skill <name>.html, and reads not-applicable in the Regression cell when there is no previous version.
Sort: any red first with regressions above efficacy failures, then fragile-but-passing, then clean green, ties broken by skill name.
The fragile tier reuses the 0006 per-badge chips: the efficacy chip now carries its win count so the leaderboard reads it against the pass floor without recomputing margins. Fragility is scoped to the passing tier so a red row never carries a chip.
SKILL.md gains the no-argument batch flow (step 7) and the index invocation.
The benchmark-core fixture test is extended to cover the tiered sort, the not-applicable Regression cell, and the per-skill links.

Deviations: the in-session orchestration half (enumerate skills, run each) is documented in SKILL.md rather than unit-tested, following the 0006 split; the benchmark harness itself was not run per the invocation. Two leaderboard fixtures (robust-skill, leaky-skill) are synthesized inline in the check because no committed bundle reaches the clean-green tier or the efficacy-red-with-fragile-regression crossing.

Review

Risk

Overall: Low

Blast radius: Low — one new CLI branch and pure functions; the existing scoring path is only refactored into a shared helper, contract preserved.
Reversibility: Low — purely additive transform, no schema or migration; plain revert.
Test coverage: Low — the Nix fixture directly exercises sort, N/A cell, fragile-axes, and links.
Sensitive domain: Low — reads JSON, writes HTML/JSON; no auth, permissions, or concurrency.
Size & complexity: Low — ~200 lines, straightforward control flow.
Runtime criticality: Low — dev-only tooling, run deliberately, no LLM.

No unaddressed Standards or Spec findings — the three actionable review findings (a cross-file comment, the duplicated HTML document skeleton, and fragility chips leaking onto red rows) were all fixed in the diff.

Task: `.claude/tasks/0007-all-skills-leaderboard.md` ## Summary Adds the all-skills leaderboard to the benchmark skill's deterministic core. - A second pure transform (`build_leaderboard` / `render_leaderboard`) plus a `--leaderboard` CLI mode that consumes the per-skill `*.results.json` files and renders `tests/.reports/index.html`. - Each row carries both verdict badges, links out to its per-skill `<name>.html`, and reads not-applicable in the Regression cell when there is no previous version. - Sort: any red first with regressions above efficacy failures, then fragile-but-passing, then clean green, ties broken by skill name. - The fragile tier reuses the 0006 per-badge chips: the efficacy chip now carries its win count so the leaderboard reads it against the pass floor without recomputing margins. Fragility is scoped to the passing tier so a red row never carries a chip. - `SKILL.md` gains the no-argument batch flow (step 7) and the index invocation. - The `benchmark-core` fixture test is extended to cover the tiered sort, the not-applicable Regression cell, and the per-skill links. Deviations: the in-session orchestration half (enumerate skills, run each) is documented in `SKILL.md` rather than unit-tested, following the 0006 split; the benchmark harness itself was not run per the invocation. Two leaderboard fixtures (`robust-skill`, `leaky-skill`) are synthesized inline in the check because no committed bundle reaches the clean-green tier or the efficacy-red-with-fragile-regression crossing. ## Review ### Risk **Overall: Low** - Blast radius: Low — one new CLI branch and pure functions; the existing scoring path is only refactored into a shared helper, contract preserved. - Reversibility: Low — purely additive transform, no schema or migration; plain revert. - Test coverage: Low — the Nix fixture directly exercises sort, N/A cell, fragile-axes, and links. - Sensitive domain: Low — reads JSON, writes HTML/JSON; no auth, permissions, or concurrency. - Size & complexity: Low — ~200 lines, straightforward control flow. - Runtime criticality: Low — dev-only tooling, run deliberately, no LLM. No unaddressed Standards or Spec findings — the three actionable review findings (a cross-file comment, the duplicated HTML document skeleton, and fragility chips leaking onto red rows) were all fixed in the diff.

alexion added 1 commit 2026-07-24 16:58:36 -04:00

feat: add all-skills leaderboard (task 0007) 97c0eeb055

Add a second pure transform to the benchmark core: a `--leaderboard` mode
that ranks per-skill results models into an index leaderboard, one row per
skill carrying both verdicts, links out to each per-skill report, and sorts
any red first (regressions above efficacy failures), then fragile-but-passing,
then clean green.

The fragile tier reuses the 0006 per-badge chips: the efficacy chip now
carries its win count so the leaderboard reads it against the pass floor
without recomputing margins, and fragility is scoped to the passing tier so a
red row never carries a chip. The runner's SKILL.md gains the no-argument
batch flow and the index invocation. The fixture test covers the tiered sort,
the not-applicable Regression cell, and the per-skill links.

alexion merged commit 97c0eeb055 into main

2026-07-24 17:03:44 -04:00

alexion deleted branch task-0007-all-skills-leaderboard

2026-07-24 17:03:44 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alexion/skills#7