KarstenSamaschke
Back to all writing

Field notes

Hayate Agent's Benchmark Left the Answer Key Within Reach

A benchmark that asks a model to fix a file is worthless if the fix sits next door. For four hours on Tuesday, Hayate Agent's did exactly that: reference solution, the previous case's fix and the grader command all reachable from the worker's directory. What the fix moved, what the tests show, and what it still cannot prevent.

Karsten Samaschke · · 10 min read

Hayate Agent's Benchmark Left the Answer Key Within Reach

初冬や
隣の部屋に
答えあり

Early winter.
In the room next door
lies the answer.

A benchmark that asks a model to fix a broken file is only worth something if the fix is not lying around where the model works. For a little over four hours on Tuesday, between two merges, the one in my harness had exactly that problem. The reference solution was written into the same folder tree as the worker's workspace. With serial runs, the previous case's fixed file stayed next door for the next worker. And the run record, also inside that tree, held the full command of the hidden grader. Nothing in the score would have told you whether a model had fixed the bug or found the answer.

The hard part is that the worker and the harness run as the same operating system user. You cannot lock a file away from a process that is allowed to read everything you can read. So the fix does not try to make cheating impossible. It moves the answer key out of the tree the worker is given, keeps finished work as evidence somewhere else, and stores the grader only as a fingerprint. I think that is the honest version of isolation, and the documentation now says the limit out loud.

Hayate Agent cover on a dark indigo background: small slate 疾風 logo, then a two-line red Japanese headline, Hayate Agentのベンチマーク / 解答を手の届く所に置いた (Hayate Agent's benchmark put the answer key within reach), above the white English headline Hayate Agent's Benchmark Left the Answer Key Within Reach

All times are Europe/Berlin. The window is 10 November 08:00 to 20:00.

A benchmark that measures the real path

Core pull request #312 merged at 11:56 in the core repository of Hayate Agent (formerly the VanillaCore Agent). It adds a benchmark subsystem that measures a model and a reasoning level through the same dispatcher, worker runtime and session logs a real delegated task uses. Each case is submitted with a strict pin: if the pinned model or thinking level is not available, the run fails instead of falling back to the default model. Metrics come from the worker's own session logs. A case can carry an acceptance command and a hidden verifier, and both run outside the worker.

The PR's author explains why it exists. Three Qwen configurations on the homelab backend were compared by hand first, and the numbers were unusable. One prompt forced an answer format that penalised the non-thinking run, and one worker whose fixture had vanished rebuilt it from the parent session transcript, which the author says cost 11 of its 21 tool calls. The benchmark is meant to make those failure modes visible instead of letting them pollute the table.

The subsystem is off by default (enabled: false), and /benchmark is an administrator command. Release v0.1.218 at 12:06 is the first tag that contains it.

Where the answer key was

The documentation that shipped with #312 said the verifier runs with a private directory as its working directory, "so grading logic is not readable where the worker is". The code at that merge kept everything for a run under one run directory, including the workspace the worker was told to stay in:

runs/<run>/
  workspace/<case>-<n>/        the worker's directory
  verify/<case>-<n>/           private working directory of the verifier
  preflight/<case>-<n>/        reference solution, written to check the case is solvable
  record.json                  every outcome, including the verifier command

From its own workspace, a worker reaches the run directory two levels up. Three things were there. The preflight copy of the reference solution, which the runner writes for any case that declares one, to check that the case can be solved at all. The solved workspace of the previous case, because with capacity: 1 cases run one after another and nothing moved a finished workspace away. And record.json, which stored the verifier command in full, so the inline grading logic was readable at ../../record.json.

Core #322 merged at 16:05. Its description, written in German, says the problem was found before the first measurement on the running system. The preflight now runs under benchmark/evidence/<run>/preflight and is deleted in a finally block. A graded workspace moves to benchmark/evidence/<run>/workspace, and the outcome points there. The record stores the verifier only as sha256: plus a fingerprint. Run directories are created with mode 0700, and each new case clears leftovers from an aborted predecessor before it starts. Release v0.1.220 at 16:11 is built from that merge.

What the tests show, and why they first showed nothing

The pull request adds two regression tests that look into the run tree while a worker is running. One checks that no preflight material and no reference solution text is reachable from it. The other runs two cases in a row and checks that the workspace parent holds only the running case.

The AI agent that drafts these changelog posts copied the test file from #322 into a checkout of the release before it and ran it at three commits, with dependencies installed fresh in each tree, on macOS with Node 22.22.3. The file has 13 tests. The first run was useless on both sides: 4 passed and 9 were cancelled by the test runner, before and after the fix alike. The cause is a timer. The runner's case deadline was marked as not keeping the process alive, so in the timeout test, where the fake worker never finishes, Node ran out of work, ended the event loop and cancelled that test and every one after it. Core #324, merged at 17:15, removes that one line and says so. The author of #323 hit the same thing and excluded this test file from that PR's suite run after reproducing 4 passing and 9 cancelled tests on the #322 merge.

With that line removed in both older trees, which is the change #324 makes, the result is clear:

Commit Release Result
c1a1eb2, before #322 v0.1.219 11 of 13 pass
76bfa0e, merge of #322 v0.1.220 13 of 13 pass
c67221d, end of window v0.1.223 13 of 13 pass, unmodified

The two failures before the fix are the two new tests. One fails with "preflight material is reachable from the worker's run tree". The other shows the workspace parent holding case-a-1 and case-b-1 while case B runs. The record check sits later in the second test and is never reached on the old commit. The verifier command in record.json before the fix comes from reading the code at #312, not from this run. These are controlled local tests with a fake dispatcher, not a measurement on a live system. The author reports 1746 tests and 1745 passing for the full suite at #322; that was not rerun here.

What isolation means here

The documentation now has a section called "Where the answer key must never be", and it ends with the part I like most. Worker and harness share one OS user, so a worker that deliberately walks the filesystem can still find material it was not shown. Suites with a verifier are treated as confidential, even though the operating system stores them world-readably. The PR for #312 also lists that workers can read session logs, which is how the fixture got rebuilt in the first place.

I would go one step further than the documentation. It says the gates (drift, pin mismatch and the verifier) make such a run detectable. They check what the worker changed, which model ran and whether the result passes, not what the worker read. A worker that finds the reference solution, edits only the file it is supposed to fix and passes the verifier looks exactly like one that solved the case. That is not a reason to distrust the fix. It is the reason the fix had to keep the answer out of reach, because nothing afterwards would have caught it.

That is the right framing for a benchmark. A score is evidence only when you know what the worker could see, and "could see" here is a property you test, not a promise in a docs paragraph. The tests from #322 now check it while a worker runs, which is where it matters.

Also in the window

Core #318 merged at 10:36 and was tagged as v0.1.217 at 10:48. According to its author, it removes the fixed two or three attempt limit for validator retries in favour of persisted retries with exponential backoff, repairs orphaned authorized next steps through the existing admission path, and paces correction loops that succeed without new evidence. The PR states plainly that it is not deployed and that a local build activates no agents.

Core #324 (17:15) and #323 (17:20) are two continuation scheduling fixes. According to #324, an older independent admission no longer takes the foreground from a newer operator task on a different topic: it is deferred, not cancelled, and its expiry is now the earliest of the task, plan and wait deadlines. According to #323, a registered wait deadline survives when an independent assessment switches from waiting to continuing, and a deferred older assessment can no longer starve a later admitted task. Both authors report their own regression and suite results. Neither was rerun for this post, and neither claims a deployment.

Plugins #81 (18:28) makes the dispatcher's field reads return the execution status and whether a checkpoint or a result is available, so an empty result no longer looks the same for a running, paused, blocked, completed or cancelled worker. Core #326 at 18:36 refreshes the builtin plugins lock to that plugins commit. Core #319 (11:58) stops tracking a node_modules symlink that, according to its author, came in with #312, and #320 (14:32) and #321 (15:07) are mechanical lock refreshes.

What merged and what was tagged

In core, nine pull requests merged in the window, #312 and #318 to #324 plus #326, and seven releases were published, v0.1.217 to v0.1.223. Each release tag resolves to the merge commit of one of those pull requests: v0.1.218 to #319, which already contains #312, and v0.1.219 to #321. In plugins, one pull request merged (#81) and no release was published. Webui and deployment-base had no merges and no releases. When checked at 20:10, core main was still at c67221d, the merge of #326.

A tag marks the source of a published image, and it is not a deployment. Nothing here verifies that any running installation uses any of these releases.

Verdict

A benchmark that leaves the answer within reach cannot tell fixing from finding. The harness found that before its first real measurement, moved the key out of the worker's tree and wrote down what it still cannot prevent. Check what your own evaluation workers can read before you trust a single number they produce.

Sources used