KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

The Agent Did Its Research, and the Harness Called It Stuck

An agent on the VanillaCore Agent Harness that only reads and verifies used to be stopped as stuck after twenty turns, because no tracked file changed. Three changes merged on Saturday let a new milestone from an independent check, or a set of tool results the task has never seen, count as progress, while an ordinary tool poll that keeps returning the same result still hits the same limit.

Karsten Samaschke · · 9 Min. Lesezeit

The Agent Did Its Research, and the Harness Called It Stuck

Picture an agent on the VanillaCore Agent Harness with a job that is mostly reading: check thirty documents against a specification, or verify a list of claims one by one. Every turn finds something new, and no turn changes a file in the repository. Until Saturday afternoon, that job had a hard end. The Harness counted twenty turns in a row without a new state of the tracked files as twenty stalled attempts and blocked the task, so honest research and a stuck loop looked exactly the same from the outside.

I wrote about the Friday change in The Harness Step Limit Counted the Wrong Thing, and its blind spot was documented right in the code: research, remote calls and a passing test run on unchanged code all counted as repeats. Three changes merged on Saturday afternoon close most of that gap. I think they do it the right way, because the agent still cannot declare its own progress; something outside the working model has to see it. The twenty-attempt limit stays, the evidence that resets it got wider, and an ordinary tool poll that keeps returning the same result still runs into the wall.

Cover photo of a stack of blank papers with a magnifying glass on a dark desk, with the hook: Research that changed no files counted as stuck after 20 turns. Now fresh results count.

Why the limit exists at all

A task in the Harness can keep going on its own: the agent finishes a turn, plans the next step and the Harness queues it. That is useful for long jobs and dangerous for broken ones, because an agent that retries the same failing command can burn tokens all night. Until Saturday, the Harness therefore stopped a task after twenty consecutive attempts without evidence of progress, and the only evidence it accepted was a new fingerprint of the files Git tracks in the agent's working directory. That rule is strict and ignores everything the model says, although a new tracked file state can count whether it was useful or not, and it fits a coding agent well. It does not fit an agent whose real output is a finding, a verified check or an answer.

Two new kinds of evidence

Core pull request #185, together with plugins pull request #30, uses a component the Harness already had: the completion assessor. After an eligible successful working turn, as long as the task still has allowance, time and authorization left and nothing is paused, stopping or queued for that session, a separate model call with no tools and no authority to approve work classifies whether the task is done, waiting, blocked or should continue. The plugin change teaches that assessor to optionally report progress as well: a flag and a short, stable name for the milestone the turn reached, such as a verified requirement or a research conclusion. Its instructions say explicitly that a checker call, a nudge, changed wording, a timestamp or an unchanged status poll is not progress, and that the name must stay the same when the same result is retried or rephrased.

The Harness hashes that name and stores it in the same capped history that already holds the file fingerprints. A new milestone resets the stall counter; a milestone the task has already seen does not, even after a restart or an admin reset. The working model cannot use this path. If the agent's own continuation plan contains a progress claim, core ignores it, and a regression test drives an agent that claims fake progress on every turn into the limit after twenty turns.

Core pull request #186, merged about an hour later, covers the case without Git and without an explicit verdict. A successful turn that used work tools now gets a receipt when every one of those tools finished without error and returned a result: one hash over the turn's ordered pairs of tool name and result text. A receipt the task has not seen before resets the stall counter; an identical receipt does not, and a turn with a failed, empty or unfinished tool gets none. The Harness's own control tools for continuation and context maintenance do not count as work. A third rule covers turns that only think: when the assessor says "continue" without a progress verdict and the turn used no work tools, that turn neither consumes the allowance nor clears earlier stalls.

What the regression tests show

The two pull requests ship sixteen regression tests between them. To see the change as behavior rather than as a diff, the AI agent that drafts these changelog posts ran those same tests against three commits: before both changes (v0.1.119), after #185 (v0.1.120) and after #186 (v0.1.121). The tests replace the model with a scripted fake and the chat with an in-memory channel, so this is a controlled local run, not a live agent.

Commit Tests passing
v0.1.119, before both changes 8 of 16
v0.1.120, after #185 12 of 16
v0.1.121, after #186 16 of 16

Two of those tests make the difference concrete. In the first, the test provides no Git evidence at all, and on every turn the scripted agent's read tool returns a different result, for 35 turns. Before Saturday's changes, the task stopped after 20 turns. At v0.1.121 it runs all 35 turns it was scripted for. In the second, the agent checks CI on every turn, the result is always "CI still running", and the agent writes a new plan each time. That task stops after at most 21 turns at all three commits, which is the part that must not change: a poll that keeps returning the same thing is still a stall, however creatively the agent rephrases its next step.

What still counts as stuck, and what slips through

The limits are written into the documentation, and they matter. The assessor is itself a model, so its milestone names are "advisory semantic evidence, not proof of usefulness or an authorization grant". An assessor that invents a new name for the same problem on every turn would reset the counter each time; the stable-name rule and the deduplication reduce that risk but cannot rule it out. The receipt path has a similar hole, and the documentation admits it: a tool result that contains a changing timestamp can look new even when nothing else changed, which is why pure status polling is supposed to use the separate wait mechanism with its fixed deadline. Receipts prove that a result was observed, not that it was useful, and they give no permission for remote effects.

The bounds that make this tolerable are unchanged. The evidence history still holds at most 256 entries per task, and once it is full, nothing earns progress credit anymore. Deadlines are never extended by progress, the original mandate still applies, and no new turn ceiling replaces the old one. Core #185 and plugins #30 both state that cross-family independence of their reviews was not established, so the review could not be shown to come from a different model family, and #186 notes an open follow-up for malformed progress data that it deliberately did not bundle.

The rest of the day

The continuation work started in the morning. Core #175 and plugins #29 let the assessor continue a task when the model stops with a wait that is not backed by a real dependency, ask for context compaction or a handover before continuing when the context is nearly full, and remove a separate cap on total depth. Core #184 clears a stale recovery reservation once a valid continuation exists, and #182 applies a valid final verdict from the assessor to turns the model itself had marked as blocked.

Five other merges are unrelated to continuation. #176 limits how many processes and threads the targeted test runner starts. #179 and #180 let a writable local settings file change a narrow set of runtime settings, dispatcher rules and model opt-ins, for future calls, and ignore invalid or forbidden values instead of failing at startup or reload. #181 adds an authenticated, expiring admission lease for updates on a native host. #183 corrects two regression tests and changes no production code.

What merged and what shipped

All times are Europe/Berlin. The window runs from 08:00 to 20:00 on Saturday, 31 October 2026, and covers the three public repositories the harness organization listed when checked: core, deployment-base and plugins. Core merged ten pull requests between 11:35 and 18:02, plugins merged #29 and #30, and deployment-base merged nothing.

Core published eight release tags in the window, v0.1.114 to v0.1.121. Resolving the tags to commits shows v0.1.119 at the merge of #184, v0.1.120 at the merge of #185 and v0.1.121 at the merge of #186. Core pins the plugins repository by revision, and from #185 on, that pin is the merge of plugins #30. A release tag is not a running agent, so this post makes no claim about which installation runs any of this.

Where this lands

The Friday version of the limit trusted only the file system, which made it strict and simple and wrong for a longer job that needs more than twenty attempts in a row without touching a tracked file and ends in a finding instead of a diff. Saturday's version accepts two more kinds of evidence, a milestone named by an independent check and a set of tool results the task has never seen. Neither is a progress claim the working model can write into its own plan, although the agent does choose which tools it calls, and that is exactly where the timestamp hole sits. I would rather have a limit with two documented holes than one that blocks research for doing research, as long as the holes stay written down and the twenty-attempt wall stays where it is.

Sources used