KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

The Night Shift That Missed Its Start Bell

A five-minute timer could find the start of my Hayate Agent's night shift lost and still not start it. PR #195 lets that check retry a missed start, only inside the authorized window and with the same turn budget, and the merge's new tests fail before it and pass after.

Karsten Samaschke · · 8 Min. Lesezeit

The Night Shift That Missed Its Start Bell

冬隣
鐘聞き逃し
また五分

Winter is near.
The bell rang, not heard;
five minutes, once more.

My Hayate Agent (formerly the VanillaCore Agent) has a night shift. Inside a window I configure, it runs improvement experiments on its own code with a fixed turn budget and reports back after the window ends. Until this morning, a timer started that shift at the first hour of the window, and a second timer checked the controller's state every five minutes without permission to start anything. If the start signal arrived while one of those five-minute checks was still running, the start was dropped, and no later check could make up for it. That night stayed empty until the next scheduled start, unless the harness happened to restart inside the window.

Pull request #195, merged this morning at 05:38, gives the five-minute check the same start permission the start timer has, under the same guards and only inside the authorized window. I think that is exactly the right size for the fix. A scheduler that loses a signal it was given is a bug. A catch-up path with more authority than the original start would be a worse one, and this change grants the check nothing the start timer did not already have.

Hayate Agent cover: small slate 疾風 logo and "Hayate Agent" at the top, the red two-line Japanese headline 始業ベルを / 聞き逃した夜勤 (the night shift that did not hear its start bell), the white English headline "The Night Shift That Missed Its Start Bell" and github.com/hayate-agent at the bottom, on an indigo-black field with a faint wave pattern along the lower edge

Where the start got lost

The night controller does its work under a single flag. While one check is in progress, any other call returns at once, so two checks never interleave. The start timer called the controller with permission to begin a night, and the five-minute timer called it without that permission, as a pure state check. When both met, the call with permission lost: it found the flag set and returned without recording that a start had been requested. The next five-minute check found no request and, lacking the permission, did nothing. The end-of-window timer fires when the window has already closed, and a closed window starts nothing.

A failing scheduled call left no trace either. Before the fix, errors in these timer callbacks were swallowed without a log line.

The PR text is careful about what this explains. Its author reproduced the lost start in a test and writes that the association with either of the historical nights without work remains unknown. I read that as a real race found while looking for the cause of empty nights, and not yet proven to be that cause.

What the fix changes

Since #195, the five-minute timer calls the controller with the same start permission as the start timer. It does not know whether a start was dropped; it starts the current night if that night is eligible and has not started yet. Every guard the start already had still applies: the policy must be enabled, the clock must be inside the window, a nightly strategy must be selected and not in the middle of a change, and an existing night keeps its identity and its used turn budget, so a retry cannot reset a night that already ran. If the harness is paused or another conversation is busy, the night is recorded but no turn is queued. The window, the budget and the policy are unchanged, and the PR adds no replay and no manual start.

Scheduled-call failures now leave one log line with a coarse class (syntax, type or other failure), at most once every five minutes. The error message itself is never logged, because the code comment notes that policy and parser errors may contain secrets.

What a scripted run shows

The AI agent that drafts these changelog posts ran each merge's new tests against the code on both sides of that merge, with Node 22 on a local machine. The tests fire captured timer callbacks by hand against a temporary state directory and a test harness; they do not run a live system.

Change Tests run Before the merge After the merge
#195 night-start catch-up (before c7e6dc0, after a14f0d2) 4, new test file 1 pass, 3 fail 4 pass
#241 MCP progress privacy (before f0b2375, after c7e6dc0) 59, progress and Matrix test files 57 pass, 2 fail 59 pass
#242 topic inheritance (before a14f0d2, after 29e3cd3) 13, three topic test files 11 pass, 2 fail 13 pass

The first #195 test reproduces the race. A state check holds the flag, the 23:00 start fires into it, and the night stays at the previous date. On the old code the following five-minute check leaves it there, so the test reads 2026-10-30 where it expects 2026-10-31. On the new code the same check starts the night of 2026-10-31 and queues its first turn, and a second check changes nothing. Two more failures follow from the missing start: the pause test finds no state file, and the error test counts zero log lines where it expects one. The one test that passes on both sides is the negative case, in which no night starts outside the window, with the policy disabled, or with the strategy disabled or changing. At the last merge, 29e3cd3, all six test files together pass 76 of 76.

The PR author reports 1364 of 1364 tests passing in a full serial run and a review with zero P0/P1 findings. That is the author's report, and the run described here covered only the files above.

The other two merges

In full progress mode, the harness posts each tool step into the chat, including its details. With #241, merged at 20:42 yesterday evening, the source shows direct MCP tool calls and the MCP resource helpers show only their name and status in that stream, and their arguments and results are never read for it. The new test makes the detail field throw if anything touches it, and the Matrix test asserts that the HTTP requests sent during the test do not contain the private payload. The PR names its own limit: an MCP call nested inside codemode is not recognized, and reduced progress mode remains the way to hide every tool payload. I prefer a limit written into the PR over one a reader discovers later in a chat log.

#242, merged at 05:43, separates topic relevance from tag inheritance. When the classifier linked a new message to a related earlier topic, the old code copied every umbrella tag of that topic onto the message. Now the classifier has to state explicitly whether the message continues the whole subject, and hashtags inside an activity line copied back into a message no longer count as the operator's own tag choice. The tests check this contract with a scripted classifier, and the PR makes no claim about how well a real model judges it.

What merged and what was tagged

The window ran from 20:00 on November 3 to 08:00 on November 4, Europe/Berlin time, and all times in this post use that zone. Core merged exactly these three pull requests in it, while plugins and deployment-base merged nothing. Each merge is the target of its own release tag: v0.1.166 is the #241 merge (published 20:50), v0.1.167 the #195 merge (05:44) and v0.1.168 the #242 merge (05:50). A git ancestry check confirms that v0.1.166 does not contain #195. A release tag is not evidence of a build or a rollout: whether any running Hayate Agent uses these changes is not verified here, and #195 itself performed no restart or live activation. In the source inventory for this window, core main was at 29e3cd3, plugins main at 1b07370 and deployment-base at 9776088. The public GitHub snapshots under hayate-agent were not checked for these changes.

Verdict

An unattended night shift is only as reliable as its start. In the merged source, a start that gets lost now has a second chance every five minutes, inside the window I authorized and with the budget the night already had. That is all the authority a retry should get. Whether this race caused any of the empty nights is still open.

Sources used