Zurück zu allen Artikeln

Notizen aus der Praxis

A 30-Character Message Waited 19 Minutes

A 30-character chat message took 19 minutes to reach Hayate Agent's first tool call, most of it spent on a local model busy with other large prompts. Core #507 lets an installation limit how many requests one runtime sends to a local model at once and puts interactive requests first. A scripted run shows what that changes, and what it does not.

A 30-Character Message Waited 19 Minutes

冬ごもり
三十文字が
待ちつづけ

Winter seclusion.
Thirty characters
keep on waiting.

A fresh 30-character message to Minerva, one of the agents running on Hayate Agent, came in through a Matrix chat, and 19 minutes and 34 seconds passed before the agent made its first tool call. The two tool calls then took 5 and 1 milliseconds. The rest went to the model: the first request hit the 300-second timeout, the agent waited 2 seconds and sent it again, and the second attempt took 872 seconds for 6,694 prompt tokens and 374 output tokens. That is how the description of core pull request #507 reports the case, and it adds the reason: the local DeepSeek model was processing several large, uncached prompts at the same time and produced about 0.7 to 0.8 tokens per second. The short message had arrived at a model that was already busy with other large prompts.

The hard part is that the obvious fixes each touch only one piece. A longer timeout can spare the retry, but it does nothing about the other prompts the model is working on. Switching the agent to another model behind the operator's back changes what it does, not just how fast it does it. Cutting the context makes one prompt cheaper while the others keep competing for the same model. #507, merged at 07:00 this morning, puts a small gate in front of the model instead: an installation can set how many requests one runtime may send to a given local model at once, and when requests have to wait, the human's goes first. I like that it gives the model fewer requests at a time instead of making a single request smaller.

Hayate Agent cover on a deep indigo-black background with a wave pattern at the bottom: small slate brush logo 疾風 and 'Hayate Agent' at the top, a two-line red Japanese headline '三十文字のメッセージが / 十九分待った' (a 30-character message waited 19 minutes), the white English headline in two bold lines 'A 30-Character Message / Waited 19 Minutes', and github.com/hayate-agent at the bottom.

What a scripted run shows

The AI agent that drafts these changelog posts wrote a throwaway test that drives the real runtime against a fake local model. Four background worker sessions send a request, and 30 milliseconds later a chat session sends one. Each request needs 400 milliseconds of the fake model's time, and requests that run at the same time share that time equally. The identical file ran three times each on v0.1.346 (the release before #507) and on the last merge of the window, once without the new setting and once with concurrency: 1.

Four workers busy, one chat message Setting absent Concurrency 1
Requests at the model at once 5 1
Chat answered after 2.46 to 2.48 s 0.99 s
All five done after about 2.5 s about 2.5 s

Without the setting, all five requests reach the model together, and the chat answer arrives last, together with the workers, because it started last and got an equal share. That holds on v0.1.346 and on the window's last merge alike, so leaving the setting out changes nothing. Both times count from the moment the chat message was sent. With the setting, the first worker request that reached the model keeps running, the chat request waits about half a second for it, goes next, and the other three workers follow. The last request finishes at about the same time in both cases. The work is the same; only the order changed. These are controlled local runs with a fake model and a deliberately simple sharing rule, not a measurement of Minerva or any real backend.

How the gate works

The setting is a list in the installation's configuration, one entry per local model:

localModelScheduling:
  - model: homelab-main/deepseek-v4-flash
    concurrency: 1

The runtime identifies a model by its endpoint address and model ID, so two provider aliases that point at the same physical model share the same limit. Main turns, workers, subagents, native summaries, the HTTP classifier, planner and review calls and review worktrees in that runtime all go through it. Other models are not affected. A permit covers one complete model response, not the tool calls that follow it, so a worker running a long tool does not hold the model.

Chat turns from Matrix, the web UI, the terminal UI, Telegram, Slack, Discord and Signal count as interactive by default, and a gateway the runtime does not know counts as background unless the request asks otherwise. Dispatched work, subagents, cron, ephemeral sessions and native summaries are always background, and so are classifier, planner and review calls. Other API runs count as background unless the run asks to be treated as interactive. A waiting interactive request goes ahead of waiting background requests, but after four interactive requests in a row a waiting background request gets its turn, so a steady stream of chat messages cannot hold background work back forever. A request that is already running is never cancelled. The wait happens before the HTTP request and its timeout are created, so waiting in this queue cannot itself cause the timeout-and-retry pattern from the opening, and a request cancelled while it waits sends nothing.

In a chat with gateway progress switched on (it is off in the default configuration), a waiting request becomes a progress message, "⏳ Wartet auf lokale Modellkapazität; noch keine eigene Modellaktivität." (waiting for local model capacity, no model activity of its own yet), followed by one saying the request was admitted and the provider's progress is still unknown. The run API and the terminal UI get the same states. In the scripted run, the test recorded exactly two queue events for the chat session, waiting and admitted.

What it does not do

The PR description is explicit about its limits, and they matter for anyone who reads this as a fix for Minerva:

  • It works inside one runtime. Another agent or process using the same model is not counted, and no hardware capacity is reserved.
  • A long request that is already running is not interrupted, so a human request can still wait for it, and requests already in flight when the setting is switched on still have to finish.
  • It does not limit how long the provider takes to queue or process a prompt once it is sent.
  • Without the setting, behavior is unchanged. With a rule that names a model the runtime cannot resolve, every model request in that runtime fails, including requests for other models, instead of running unscheduled; the description lists that as a known, non-blocking review finding.
  • The description says the real latency has to be measured again after the Minerva configuration is activated, which it treats as a separately reviewed step. Nothing in the window shows that activation.

#503, merged at 20:38 on Sunday, explains part of the contention from the other side. Its description reports that all four occupied worker slots ran on the same DeepSeek model, although dedicated Qwen worker roles were configured, because work whose classification failed fell back to Main's model. #503 sends new unclassified work to the configured worker role instead, and states that it does not move the four workers that were already running. Together, the two changes address who uses the model and in which order. Neither one makes the model faster.

What merged and what shipped

All times are Europe/Berlin. The window runs from 20:00 on Sunday, 15 November, to 08:00 on Monday, 16 November. #507 merged at 07:00 and its merge commit is the target of release v0.1.347, published at 07:08. A release tag identifies the tagged source commit; it says nothing about which installations run it, and nothing here says the setting is enabled anywhere.

Eighteen core pull requests merged in the window, and core published six releases, v0.1.344 to v0.1.349:

  • #502 requires referenced evidence before a retry decision (v0.1.344).
  • #503 keeps unclassified work on the configured worker role, and #504 respects operator authority and coordinates worker takeover (both v0.1.345).
  • #505 retains validator uncertainty, and #500 freezes a verified loop for a safe scheduler handoff.
  • Between 01:27 and 05:23, eight smaller fixes: #508, #510, #511, #513, #514, #516, #517 and #518, several of which reject unknown or malformed configuration and stored data.
  • #520 bundles the Matrix native activity default into core, the commit tagged v0.1.346, which also contains #500, #505 and the eight fixes.
  • #507 is in v0.1.347, #506 adds a scoped owner cron reload and one-shot admission (v0.1.348), and #521 fixes truncated tag metadata in native updates (v0.1.349).
  • #522, which asks Main for unresolved completion assessments, merged at 07:58. Its merge commit is the target of release v0.1.350, published at 08:10, just after the window closed.

In plugins, #109 respects operator authority in completion guidance, #110 retains TypeSafe uncertainty diagnostics, and #111 enables native Dispatcher and Worker activity in Matrix by default. deployment-base and webui merged nothing in the window.

Sources