fix(bench): gate seeding on repo+index readiness and match read counts semantically
Two benchmark-harness races were being scored as agent failures. A freshly seeded throwaway repo could 404 for an independent reader (the agent, a fresh process) before Gitea made it consistent, and the async issue indexer lagged so `search issues`/`search prs` returned nothing right after seeding — leaving the agent unable to find the target it was asked to act on. seedRepo now blocks until an independent read confirms the repo is reachable, its full seeded issue and pull spread is visible, and a seeded issue and pull are returned by the search index, before releasing the agent. It polls up to 60s and throws a loud harness error on timeout rather than letting a propagation delay become a scored agent failure. Also add an optional `pattern` regex to RequiredFact so a read answer's count is recognised semantically rather than as a fixed phrase: "5 issues are currently open" no longer fails against the literal "5 issues are open", while the alphabetic-only filler run keeps a wrong count beside the right number (e.g. "5 issues ... 3 open") failing.
This commit is contained in:
@@ -131,10 +131,21 @@ export interface RepoState {
|
||||
* whitespace and case normalization), so a count or a name can be phrased
|
||||
* variously without resorting to an LLM judge. `description` names the fact in
|
||||
* diagnostics when it is missing.
|
||||
*
|
||||
* `anyOf` matches a *contiguous* substring, which is brittle for facts a human
|
||||
* naturally pads with filler — "5 issues are currently open" does not contain
|
||||
* the fixed phrase "5 issues are open". For those, supply `pattern`: a regular
|
||||
* expression (matched against the same normalized report) that satisfies the
|
||||
* fact when it matches, so the count itself can be recognised rather than one
|
||||
* exact wording. A fact is present when any `anyOf` rendering *or* `pattern`
|
||||
* matches; `anyOf` stays the human-readable renderings even when `pattern`
|
||||
* carries the real matcher.
|
||||
*/
|
||||
export interface RequiredFact {
|
||||
description: string;
|
||||
anyOf: string[];
|
||||
/** Optional regex (source, matched case-insensitively against the normalized report). */
|
||||
pattern?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user