KarstenSamaschke
Back to all writing

Field notes

Hayate Agents That Dial Out Did Not Notice a Dead Connection

A Hayate Agent that dials out to its dashboard could sit on a dead connection and still report itself connected. A change this morning makes the dashboard's pings the agent's pulse: no ping for 90 seconds, and the agent hangs up and reconnects, without replaying any request that was in flight.

Karsten Samaschke · · 7 min read

Hayate Agents That Dial Out Did Not Notice a Dead Connection

冬の夜
合図途絶えて
掛け直す

Winter night.
The signal breaks off,
and the call is made again.

Since Sunday, a Hayate Agent (formerly the VanillaCore Agent) behind a home router can dial out to a central dashboard over a WebSocket instead of opening an inbound port. That solved the firewall problem and exposed a quieter one. According to the pull request that merged at 06:53 this morning, long-running connections in real use ended up half-open: somewhere between the agent and the dashboard, a router or proxy had lost the connection, the dashboard had already dropped the session, and the agent's socket still said OPEN. Nothing told the agent. Its own status kept reporting "connected", while every authorized request from the dashboard failed with "Outbound agent disconnected; no work sent".

This is the failure I distrust most in agent infrastructure, because on the agent's own side it looks exactly like an idle connection. Nothing crashed, and the agent still believed it was reachable. Plugins pull request #73 gives the agent a way to notice, and it keeps the rule from Sunday intact: the reconnect never replays a dropped request.

Hayate Agent cover: small slate 疾風 logo and "Hayate Agent" at the top, the red Japanese headline in two lines, 自ら接続するHayate Agentは / 通信断に気づかなかった (Hayate Agents that connect out on their own did not notice the dead link), the white English headline "Hayate Agents That Dial Out Did Not Notice a Dead Connection" and github.com/hayate-agent at the bottom, on an indigo-black field with a faint wave pattern across the lower part

All times are Europe/Berlin. The window is 8 November 20:00 to 9 November 08:00.

Why an open socket can be a dead one

A TCP connection has no pulse of its own. If a box in the middle forgets about it, for example a home router after a restart or a proxy that drops idle connections, neither end is told. The side that sends something eventually runs into an error. The side that only waits never does.

In this setup, the dashboard was already covered. Its relay pings every connected agent every 20 seconds by default and terminates any session that did not answer the previous ping. So the dashboard noticed the dead line, closed its end and forgot the session. The agent was the waiting side. It sat on an open socket, waiting for requests that could no longer reach it, and it had no reason to reconnect.

The dashboard's ping becomes the agent's pulse

After #73, the agent treats the dashboard's pings as required evidence that the line is alive. The code comment says it in one sentence: "Server pings are required liveness evidence. TCP can remain half-open after a NAT/proxy failure; reconnect the channel, never replay work."

When the connection opens, the agent starts a 90-second deadline, and every ping from the dashboard resets it. If 90 seconds pass without a ping, the agent terminates its own socket. That runs the normal close path, which schedules a reconnect with the existing backoff of 500 milliseconds, doubling up to 30 seconds by default. With the relay's default 20-second ping interval, the agent tolerates four and a half missed intervals before it gives up on the line, so one late ping does not tear down a healthy connection. The deadline is a constructor option between 100 milliseconds and three minutes. The outbound plugin does not set it, so in plugin use it is 90 seconds.

What the reconnect does not do

The reconnect replaces the line and nothing else. The agent's enrollment and the human approval on the dashboard stay as they were, and the connection is not revoked. Requests that were pending on the dashboard when the line died have already failed there with "Agent disconnected; outcome unknown. No automatic retry", and the new connection does not replay them. Whether such a request ran on the agent stays something a person reconciles, which is the honest answer when nobody knows.

The deadline timer also stays out of the agent's real work. According to the pull request, it does not count as foreground work, so an idle connection does not hold up a host update, and stopping the agent clears it.

The new test against the old code

The pull request adds two TLS tests. The first one makes the dashboard ping far too slowly (every 10 seconds) for a 200-millisecond agent deadline, then waits for the agent to try to reconnect and checks that the connection was not revoked and that no workspace stream was created. The second one pings every 50 milliseconds against a 300-millisecond deadline and checks that an idle agent stays connected with no active requests.

The AI agent that drafts these changelog posts ran both tests five times against the code before the merge (ee3bc45, with the new test file copied in) and five times against the merge (4dafd79), with identical dependencies, on macOS with Node 22.22.3.

Test Before #73 After #73
Silent channel triggers a reconnect 0 of 5 pass 5 of 5 pass
Regular pings keep an idle channel 5 of 5 pass 5 of 5 pass

On the old code, the first test fails every time with "Condition did not become true" after about five seconds: the agent never tries to reconnect. The second test passes on both, which is its job. It guards against the new deadline killing healthy idle connections. The complete WebSocket test file passes 8 of 8 after the merge. The pull request reports nine WSS tests. The ninth needs a Core checkout and was skipped here. None of this reproduces a real router dropping a connection. It shows that the agent now reacts to missing pings, which is the signal such a drop leaves behind.

Also in the window: how the WebUI reaches GitHub

WebUI #11 (07:15) is documentation only. It describes how the standalone WebUI gets published to GitHub: a scheduled publisher runs every five minutes, requires native CI for the exact protected source revision, and tests the public export in a separate container without credentials, including browser and real backend integration tests. An unchanged public tree is a no-op. A changed snapshot becomes a pull request on publish/webui, which an independent maintainer verifies before merging. The old manual helper stays as a recovery path only. The document also says that product deployment and the GHCR release remain separate steps. No product, workflow or browser behavior changed, and none of the described process was checked for this post.

What merged and what was tagged

In plugins, one pull request merged in the window: #73 at 06:53, 38 lines added and 7 removed across agent-link/websocket-client.mjs and its test file. The overlay release v0.4.2 followed at 06:54, and its tag resolves to the merge commit 4dafd79. In webui, #11 merged at 07:15, and main is at e139a28. Core had no merges in the window, and its main is still at a4c0a61 from 17:56 on Sunday. Deployment-base had none either.

A tag marks the source of a published overlay, and it is not a deployment. Nothing here verifies that any running agent uses v0.4.2.

Verdict

An agent that only waits for work cannot tell a dead line from a quiet evening. Now it does not have to guess: when the dashboard stops pinging for 90 seconds, the agent hangs up and dials again, and the reconnect does not replay anything that was in flight.

Sources used