alexion/gitea-axi
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(bench): gate seeding on repo+index readiness and match read counts semantically

Browse Source 
Two benchmark-harness races were being scored as agent failures. A freshly
seeded throwaway repo could 404 for an independent reader (the agent, a fresh
process) before Gitea made it consistent, and the async issue indexer lagged so
`search issues`/`search prs` returned nothing right after seeding — leaving the
agent unable to find the target it was asked to act on.

seedRepo now blocks until an independent read confirms the repo is reachable,
its full seeded issue and pull spread is visible, and a seeded issue and pull
are returned by the search index, before releasing the agent. It polls up to
60s and throws a loud harness error on timeout rather than letting a
propagation delay become a scored agent failure.

Also add an optional `pattern` regex to RequiredFact so a read answer's count is
recognised semantically rather than as a fixed phrase: "5 issues are currently
open" no longer fails against the literal "5 issues are open", while the
alphabetic-only filler run keeps a wrong count beside the right number (e.g.
"5 issues ... 3 open") failing.
This commit is contained in:
alexion
2026-07-18 23:25:32 -04:00
parent 662ba82d71
commit b0a0ffc1ab
5 changed files with 200 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
bench/checker.ts
View File

@@ -241,9 +241,7 @@ function matchByKey<T>(
*/
export function checkReadAnswer(facts: RequiredFact[], report: string): CheckResult {
const haystack = normalizeText(report);
const missing = facts.filter(
(fact) => !fact.anyOf.some((rendering) => haystack.includes(normalizeText(rendering))),
);
const missing = facts.filter((fact) => !factPresent(fact, haystack));
if (missing.length === 0) {
return { pass: true };
}
@@ -253,6 +251,20 @@ export function checkReadAnswer(facts: RequiredFact[], report: string): CheckRes
};
}
/**
* Whether a required fact is present in the normalized report: any `anyOf`
* rendering as a contiguous substring, or the optional `pattern` regex matching.
* The pattern is compiled case-insensitively over the already-normalized text, so
* it recognises a count padded with filler ("5 issues are currently open") that no
* fixed phrase would.
*/
function factPresent(fact: RequiredFact, haystack: string): boolean {
if (fact.anyOf.some((rendering) => haystack.includes(normalizeText(rendering)))) {
return true;
}
return fact.pattern !== undefined && new RegExp(fact.pattern, "i").test(haystack);
}
/**
* Lower-case, drop markdown emphasis/code markers, and collapse runs of
* whitespace so incidental phrasing and formatting do not matter — a report that