alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add benchmark seed provisioning (task 0025) #26

Merged
alexion merged 1 commits from task-0025-bench-seed-provisioning into main 2026-07-16 08:23:15 -04:00
Conversation 0 Commits 1 Files Changed 11 +980 -8
alexion commented 2026-07-16 08:08:10 -04:00
Owner
Copy Link

Task: .claude/tasks/0025-bench-seed-provisioning.md

Summary

Adds the deterministic, idempotent seed that brings a freshly provisioned throwaway repository to a known ground truth before a trial runs, scripted over the live Gitea API.

  • bench/seed-plan.ts — the pure ground truth (fixed labels, an eight-issue spread across the single-user-seed discriminating dimensions of label / state / assignee presence / title keyword, and three pull requests: labelled, reviewed, and real-branch-backed) plus groundTruth(user), which realizes it into the RepoState the checker scores against, with the shared issue/pull-request numbering a fresh repo hands out.
  • bench/seed.ts — idempotent seeding reconciled by natural key (label by name, issue/PR by title, comment/review by body, branch by name), reusing gitea-axi's own tea-login credential discovery via resolveBenchAccess (no new secret handling; selectLogin was exported from src/context.ts for this).
  • Live smoke tier (vitest.bench-smoke.config.ts, npm run test:bench:smoke) — validates provisioning + seeding + idempotency end-to-end against a real host, gated on GITEA_AXI_BENCH_LOGIN and skipping cleanly when unset, kept out of the deterministic test:bench tier. Validated live against git.alexion.dev.

Deviations (full detail in the task file's Implementation Notes): deleteRepo and readSeedSummary were added as bounded smoke-test support; pull-request state reconciliation (reopen a drifted-closed PR) was added in response to review; the ADR's self-review-promotion verification is deferred to the task-suite slice (task 0028), since the seed needs only always-permitted comment reviews.

Review

Risk

Overall: Medium

  • Blast radius — Low. All new code lives in bench/ (dev-only, npm-excluded); the sole src/ touch is widening selectLogin from private to export, a purely additive visibility change.
  • Reversibility — Low. Nothing but new files plus one export and config-list additions; no migrations, deletions, or published-API/schema changes. Trivial to revert.
  • Test coverage — Medium. The pure seed-plan is well-covered; the imperative seed.ts is validated only by a live smoke test that skips (as a pass) whenever GITEA_AXI_BENCH_LOGIN is unset, so it runs unverified in normal CI.
  • Sensitive domain — Medium. Handles auth tokens (reuses tea-login credential discovery) and creates/deletes repos over the live API; scope confined to throwaway repos, no new secret handling.
  • Size & complexity — Medium. ~900 added lines with straightforward, mostly linear reconcile-by-natural-key control flow; large but not tangled.
  • Runtime criticality — Low. Dev-only benchmarking tooling, excluded from the published package.

Standards findings left unaddressed

  • Duplicated authenticated-round-trip shape (bench/seed.ts request/send vs test/e2e/provision.ts apiRequest). Kept separate deliberately: the spec keeps bench vocabulary and code apart from test/, and the bench copy adds a JSON-returning send<T> layer, so extracting a shared client would cross that boundary.

Spec findings left unaddressed

  • Idempotency drift-restoration is under-tested. The smoke test exercises re-running the seed on a seed-produced repository (the literal AC-3 property) but not state restoration after an external mutation — the harness provisions a fresh repository per trial and never re-seeds an externally-mutated one, so this is beyond AC-3.
  • Self-review promotion verification deferred. ADR 0015 pins the "does the host permit self-approve/request-changes" check to implementation; it governs the two review tasks, so it belongs to the task-suite slice (task 0028). The seed already models all three review kinds via REVIEW_EVENT and needs only always-permitted comment reviews.
  • deleteRepo exceeds the seed's stated surface. Retained as best-effort smoke-test cleanup so the live run does not litter the host; cell-loop teardown remains the run loop's job.
Task: `.claude/tasks/0025-bench-seed-provisioning.md` ## Summary Adds the deterministic, idempotent seed that brings a freshly provisioned throwaway repository to a known ground truth before a trial runs, scripted over the live Gitea API. - **`bench/seed-plan.ts`** — the pure ground truth (fixed labels, an eight-issue spread across the single-user-seed discriminating dimensions of label / state / assignee presence / title keyword, and three pull requests: labelled, reviewed, and real-branch-backed) plus `groundTruth(user)`, which realizes it into the `RepoState` the checker scores against, with the shared issue/pull-request numbering a fresh repo hands out. - **`bench/seed.ts`** — idempotent seeding reconciled by natural key (label by name, issue/PR by title, comment/review by body, branch by name), reusing gitea-axi's own tea-login credential discovery via `resolveBenchAccess` (no new secret handling; `selectLogin` was exported from `src/context.ts` for this). - **Live smoke tier** (`vitest.bench-smoke.config.ts`, `npm run test:bench:smoke`) — validates provisioning + seeding + idempotency end-to-end against a real host, gated on `GITEA_AXI_BENCH_LOGIN` and skipping cleanly when unset, kept out of the deterministic `test:bench` tier. Validated live against `git.alexion.dev`. Deviations (full detail in the task file's Implementation Notes): `deleteRepo` and `readSeedSummary` were added as bounded smoke-test support; pull-request state reconciliation (reopen a drifted-closed PR) was added in response to review; the ADR's self-review-promotion verification is deferred to the task-suite slice (task 0028), since the seed needs only always-permitted comment reviews. ## Review ### Risk **Overall: Medium** - Blast radius — Low. All new code lives in `bench/` (dev-only, npm-excluded); the sole `src/` touch is widening `selectLogin` from private to `export`, a purely additive visibility change. - Reversibility — Low. Nothing but new files plus one export and config-list additions; no migrations, deletions, or published-API/schema changes. Trivial to revert. - Test coverage — Medium. The pure seed-plan is well-covered; the imperative `seed.ts` is validated only by a live smoke test that skips (as a pass) whenever `GITEA_AXI_BENCH_LOGIN` is unset, so it runs unverified in normal CI. - Sensitive domain — Medium. Handles auth tokens (reuses tea-login credential discovery) and creates/deletes repos over the live API; scope confined to throwaway repos, no new secret handling. - Size & complexity — Medium. ~900 added lines with straightforward, mostly linear reconcile-by-natural-key control flow; large but not tangled. - Runtime criticality — Low. Dev-only benchmarking tooling, excluded from the published package. ### Standards findings left unaddressed - **Duplicated authenticated-round-trip shape** (`bench/seed.ts` `request`/`send` vs `test/e2e/provision.ts` `apiRequest`). Kept separate deliberately: the spec keeps bench vocabulary and code apart from `test/`, and the bench copy adds a JSON-returning `send<T>` layer, so extracting a shared client would cross that boundary. ### Spec findings left unaddressed - **Idempotency drift-restoration is under-tested.** The smoke test exercises re-running the seed on a seed-produced repository (the literal AC-3 property) but not state restoration after an external mutation — the harness provisions a fresh repository per trial and never re-seeds an externally-mutated one, so this is beyond AC-3. - **Self-review promotion verification deferred.** ADR 0015 pins the "does the host permit self-approve/request-changes" check to implementation; it governs the two review *tasks*, so it belongs to the task-suite slice (task 0028). The seed already models all three review kinds via `REVIEW_EVENT` and needs only always-permitted comment reviews. - **`deleteRepo` exceeds the seed's stated surface.** Retained as best-effort smoke-test cleanup so the live run does not litter the host; cell-loop teardown remains the run loop's job.
alexion added 1 commit 2026-07-16 08:08:11 -04:00
feat: add benchmark seed provisioning (task 0025)
All checks were successful
CI / test (pull_request) Successful in 52s
Details
CI / test (push) Successful in 49s
Details
8666c53557 
Add the deterministic, idempotent seed that brings a freshly provisioned
throwaway repository to a known ground truth before a trial runs, scripted
over the live Gitea API.

- bench/seed-plan.ts: the pure ground truth (fixed labels, an issue spread
  across the discriminating dimensions, and labelled/reviewed/real-branch
  pull requests) plus groundTruth(user), realizing it into a RepoState.
- bench/seed.ts: idempotent seeding reconciled by natural key, reusing
  gitea-axi's own tea-login credential discovery (no new secret handling).
- A live smoke tier (test:bench:smoke) validating the seed end-to-end and
  skipping cleanly when no host is configured, kept out of the deterministic
  bench tier.

Export selectLogin from src/context.ts so the bench reuses the exact
credential-selection path.
alexion merged commit 8666c53557 into main 2026-07-16 08:23:15 -04:00
alexion deleted branch task-0025-bench-seed-provisioning 2026-07-16 08:23:15 -04:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
alexion
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: alexion/gitea-axi#26
Delete Branch "task-0025-bench-seed-provisioning"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?