KarstenSamaschke
Back to all writing

Field notes

Hayate Agent Ignored Every Photo Sent to It on Matrix

Until Saturday, Hayate Agent dropped every image and file sent to it on Matrix. Release v0.1.206 receives and sends them, also in end-to-end encrypted rooms, and fails closed instead of falling back to plaintext. What the gateway enforces, what it leaves to the agent, and what a scripted before/after run shows.

Karsten Samaschke · · 10 min read

Hayate Agent Ignored Every Photo Sent to It on Matrix

冬の宵
写真の鍵は
文の中

Winter evening.
The photo's key
is inside the letter.

Say you send Hayate Agent (formerly the VanillaCore Agent) a screenshot of an error in a Matrix chat and wait for an answer. Up to release v0.1.205, nothing came back. The Matrix gateway read the event, saw that it was not a text message and returned, and the line that did it carried a comment: // media later. Photos, PDFs, voice notes and videos all ended there, in plain rooms and in encrypted ones. On Saturday morning that line went away, and the agent now receives and sends images and files on Matrix, including in end-to-end encrypted rooms.

Receiving a file is the easy half. The hard half is everything around it: a file is not text, so it must not be treated as a command or batched with other messages, and in an encrypted room the obvious shortcut, sending the file unencrypted when encryption is not available, would quietly break the one promise that room makes. I like the answer the change gives. The agent sends a file only when it writes one deliberate line for it, and the gateway would rather fail the delivery than downgrade an encrypted room to plaintext. Plugins pull request #62 brought the code, and it was pinned into core with pull request #303, released as v0.1.206.

Hayate Agent cover: small slate 疾風 logo and "Hayate Agent" at the top, the red Japanese headline in two lines, Hayate AgentはMatrixで / 届く写真を全て無視していた (Hayate Agent ignored all photos arriving on Matrix), the white English headline "Hayate Agent Ignored Every Photo Sent to It on Matrix" and github.com/hayate-agent at the bottom, on an indigo-black field with a faint wave pattern along the lower edge

All times are Europe/Berlin. The window is Saturday 7 November, 08:00 to 20:00.

A file has to be asked for, line by line

For the way out, the gateway does not guess. It does not scan the answer for Markdown links or file paths. The agent has to put a line of its own into its final reply, consisting of MEDIA: followed by an absolute path, one line per file. The gateway removes that line, uploads the file into the same room and thread, and delivers the remaining text as usual. Images with the extensions png, jpg, jpeg, gif and webp go out as Matrix images, and everything else goes out as a file.

In practice the parser is stricter than that sentence suggests, and the tests check the edges. A MEDIA: line inside a fenced code block is not uploaded, so an agent explaining the protocol does not send a file by accident. Progress messages, the editable status lines the agent shows while it works, never upload anything. A missing file, a path that is not a regular file and a file above the size limit are permanent delivery failures, so the reply does not quietly arrive without its attachment.

Incoming files get the same distrust. Downloads happen only after the existing sender, room, mention and thread rules have admitted the message, and only through the configured homeserver's authenticated media endpoint. The access token is never sent to whatever server an attachment address names, redirects are refused, and a homeserver with authenticated media (Matrix 1.11 or later) is required, with no fallback to the old unauthenticated download. The file is stored with mode 0600 in a folder the gateway creates with mode 0700 (an existing folder set via MATRIX_MEDIA_DIR keeps its own permissions), and the agent receives a local path plus the original metadata, labelled as untrusted. A file named /pause does not pause anything, because attachment names are never read as slash commands.

What the gateway checks, and what it only tells the agent

The gateway enforces room admission, the regular-file check, the size limit and a 60-second transfer timeout. It does not check which file a MEDIA: line points at. The rule that the agent may only upload files the user authorized for the whole room lives in the session's capability context, as an instruction to the model: "never upload secrets or files without user authorization." That is a request, not a sandbox, and I would rather say it plainly than let a careful protocol look like a permission system. If the agent can read a file and decides to name it, the gateway sends it to the room.

Even the local media folder comes with documented limits. It is not encrypted at rest, and it has no automatic expiry, because queued or replayed messages may still point at those paths. By default each file may be up to 20 MiB, and the documentation adds that this is no quota for the folder as a whole, so disk space is the operator's job.

No plaintext where encryption is expected

In encrypted rooms, attachments use Matrix's version 2 attachment encryption: the file is encrypted with AES-256-CTR under a fresh random key, and that key, together with a SHA-256 hash of the ciphertext, travels inside the message, which is itself Megolm-encrypted. That is the haiku's key inside the letter. The homeserver stores only ciphertext under the generic type application/octet-stream, and the upload address carries no filename. Incoming encrypted files are checked against that hash, and a tampered file is rejected.

If end-to-end encryption is unavailable in that session, an upload to an encrypted room fails before any file bytes are sent. A later failure, for example while encrypting the message itself, can leave an already uploaded ciphertext unreferenced on the homeserver, but the file never goes out in plaintext. On the incoming side, plaintext messages in an encrypted room stay rejected by the existing room policy, attachments included. An encrypted message whose sending client attached the file unencrypted is still accepted and downloaded, and the documentation does not mention that case.

What a scripted run shows

The AI agent that drafts these changelog posts ran the three test files that pull request #303 adds or changes against the previous release, v0.1.205 (commit e487906), and against the merge commit 24e3d79, which is v0.1.206. Both trees were built the same way, from their own pinned plugin lock. Node 22.22.3, local temporary folders, no homeserver and no live agent.

Tests v0.1.205 v0.1.206
Attachment tests (5) 0 of 5 pass 5 of 5 pass
Encrypted channel test (1) fails passes
Capability context (4) 3 of 4 pass 4 of 4 pass

The five attachment tests cover permissions, size bounds, threads, the code-fence and progress rules, and the encrypted roundtrip with tamper detection. They use a simulated homeserver and replace the Megolm layer with a stub. The single encrypted channel test runs the real Olm/Megolm crypto machine for two devices against a simulated homeserver, and the new part of it sends a file from the agent, decrypts it on the other side and sends it back. On the old release, the failing capability test is the one that looks for the new MEDIA: instruction. At the merge, all ten pass. These are controlled local tests and say nothing about a live homeserver. The pull request author reports the full core suite at 1,606 tests with 1,605 passing, one existing optional skip and no failures; that suite was not rerun for this post.

Conversations no longer need a project

The second story of the day sits in the browser interface. At 09:10, plugins #58 started filing existing WebUI conversations under an automatically created project called "Existing chats". At 13:12, plugins #63 stopped creating that project for newly attached conversations. A conversation can now exist without any project, running in the host's working directory with the global settings. Moving it into a project or out of one changes where it is listed, and the code comment says what it does not change: "never silently retarget execution cwd." The message history stays where it was, and reading or organizing a conversation no longer moves it to the top of the list. AgentLink, the connection between two Hayate Agent instances, gained a move action that keeps the target's existing project restrictions, and conversations without a project are shared only when the target sets shareStandalone: true.

The matching interface change is webui #2, merged a few seconds later. It removes the separate Projects and Chats modes and shows one searchable list with a single agent selector in the sidebar. Conversations are still grouped by project, with the ones without a project in their own section, and each group is sorted newest message first. At 11:04 the same day, plugins #61 had made project mode the default with separate navigation for both. Two hours later that split was gone, and reverting a navigation decision on the same day is cheaper than teaching users a mode switch first. Both pull requests report their own tests, and those were not rerun for this post.

What merged and what was tagged

In core, four pull requests merged in the window. #300 (08:32) adds immutable execution contexts per conversation stream and trusted backend services, and #301 (08:52) ships a default review adapter that is aware of those contexts. #302 (11:03) lets the durable file writer store binary data, which the attachment store needs. #303 (11:09) pins plugins #62 and adds the documentation and tests. Releases v0.1.203 to v0.1.206 were published at 08:37, 08:59, 11:10 and 11:16, and each tag resolves to the merge commit of #300, #301, #302 and #303 in that order. At 20:05, just after the window, core main was at 24e3d79, a merge committed at 11:09, and a fresh fetch at 20:41 still showed it.

In plugins, eight pull requests merged, from #56 at 08:33 to #63 at 13:12. Besides #62 and #63 they cover the workspace overlay with authenticated WebUI and agent connections (#56), a federated project dashboard (#57), attaching existing chats to projects (#58), pairing target agents by slash command (#59), readable sidebar labels (#60) and the agent selector with account settings (#61). The overlay releases v0.2.0 to v0.2.4 were published between 09:10 and 13:13, and v0.2.4 resolves to the merge commit of #63. At 20:04, plugins main was at ab45982 (committed 13:12). In webui, #1 merged at 12:46, which adds independent CI and a scoped GitHub publication workflow, and #2. At 20:04 its main was at 500975b (committed 13:12), and deployment-base had no merges in the window.

A tag marks the source of a published image, and it is not a deployment. In the description of #303, the author writes that the operator asked for a rollout, and nothing here verifies that any running agent has the new gateway. The public snapshot at github.com/hayate-agent/core still named v0.1.200 as its source at 20:20 on Saturday, so none of this was on GitHub yet.

Verdict

An agent that lives in a chat and cannot see a screenshot is half deaf, and this release fixes that on Matrix. What matters more to me is the shape of the fix: files leave only on an explicit line, encrypted rooms fail closed instead of falling back to plaintext, and the documentation says openly that choosing which file to send is still the agent's call.

Sources used