KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

The Harness Step Limit Counted the Wrong Thing

The VanillaCore Agent Harness stopped every longer job after twenty follow-up steps, whether each step changed code or repeated the same failed command. A bigger number would only let a stuck loop run longer. A core change merged Friday counts only the turns in a row that leave no new trace in the tracked files, and three others let an admin restart a job blocked by its budget, the last one with a plain chat sentence. The new count is merged but not released, and no running agent is verified to have it.

Karsten Samaschke · · 12 Min. Lesezeit

The Harness Step Limit Counted the Wrong Thing

Say an agent on the VanillaCore Agent Harness gets a longer job in the chat, a refactoring that takes thirty small steps, and every step changes code in the repository. Until a merge on Friday afternoon, the Harness stopped that job after twenty follow-up steps, no matter what those steps did. A job that edited a different part of the code on every turn hit the same wall as a job that repeated one failed command twenty times. From the outside, both looked identical: the job was blocked, and a notice in the chat said the step budget was used up.

A cap like that exists for a good reason, because an agent that keeps going on its own can loop forever and burn tokens doing nothing. The lazy fix is a bigger number, and it is the wrong one, since forty would let a stuck loop run twice as long as twenty. The better fix is a cap that counts the right thing. On Friday, core pull request #165 changed the count from "follow-up steps" to "turns in a row that left no new trace in the tracked files". Three other changes that day gave admins ways to restart a job blocked by its budget without retyping it. The new count is merged but not in a release yet, which the last section covers.

A black mechanical tally counter with a thin light-blue edge lies next to a closed dark notebook on a dark navy surface, under the text: The Harness stopped jobs at step 20, even when every step changed code. The fix counts only repeats.

What counted before

A task in the Harness carries a number called depth. Every time the agent finishes a turn and plans another step, the Harness queues the next step and adds one to that number. Depth is also part of the step's identity in the queue. Before #165, the same number doubled as the budget: when a turn at depth 20 asked for another productive step, the Harness blocked the task, marked it as an exhausted productive budget and sent a notice to the conversation. From #161 on, an admin reset set depth back to zero under a fresh queue epoch, so the two jobs of that number were already pulling against each other.

That design treats every step as equally suspicious. The Harness had no idea whether a turn had moved the work forward, so it charged progress and stagnation at the same rate. The PR's own description puts it plainly: the baseline "incorrectly blocked 21 distinct steps".

What counts now

Since #165, the Harness keeps a separate repetition record next to depth. Before and after each productive turn, it takes a fingerprint of the files Git tracks in the agent's working directory: a SHA-256 over every tracked file's name and bytes. If the fingerprint after the turn is new, the turn counts as progress and the repetition counter goes back to zero. If the fingerprint is unchanged, or matches a state the task has already seen, the counter goes up by one. Twenty repetitions in a row block the task, counting the first turn, and depth now only goes up. The notice, which the Harness writes in German, now says that 20 repetitions without new host evidence are used up, instead of 20 continuation steps.

What the step limit counts, core pull request 165, merged, not released. Before: every productive follow-up step counts, no productive step 21, progress counts too. After: only turns with no new file state count, the task stops at 20 in a row, an unseen file state sets the count back to 0. Evidence comes from files tracked in Git, not model text; research, remote effects, overlapping turns and states beyond 256 earn no credit.

The details matter more than the headline number. What the agent says it did earns nothing: the Harness ignores the model's text, its tool arguments and its tool output, and reads only the files on disk. A change that is undone later does not earn credit twice, because returning to an earlier fingerprint counts as a repeat. When two turns run at the same time in one Harness, neither gets credit, because the Harness cannot tell whose change it saw. When the fingerprint cannot be taken at all, for example outside a Git repository, the turn counts as a repeat. While a task waits on a dependency, pure status polling that the Harness itself recognizes, the dispatcher's status action and continuation checkpoints, stays outside this counter and keeps the wait's fixed deadline. A recheck through the shell, the web or another plugin is charged like any other turn.

Getting a blocked task going again

Before Friday, a task blocked by its budget had no way back. The notice asked for a new explicit instruction, and that instruction became a separate task, which meant restating the job, while the old one stayed blocked. Three pull requests changed that.

#161 added an admin command, /task-budget, that inspects a task blocked by its budget and returns a token for its exact current state. A reset takes that token and a unique request ID; a separate resume afterwards needs a fresh inspection token and another request ID. The token makes sure the admin resets the task they inspected, not a version that changed in between, and the request ID makes a retried command harmless. #162 added a combined reset-and-resume, extended it to one older checkpoint shape and rewrote the notice: it now names the original task, the step that was next, why work paused and the exact commands to type.

#164 removed the need for the command syntax. An admin can now write one plain sentence in the chat, in English or German, such as "Reset the task budget for task 7 and continue". The Harness only accepts it as the whole message from an authenticated admin channel or the local console; a quoted copy inside a longer message, text from a tool result or anything the model writes does not match. Without a task ID, the Harness picks the task only when exactly one is blocked by an exhausted step budget, and asks for the ID otherwise. It saves the grant before it acts, runs the reset at the next idle moment and checks again that the sender is still an admin; a grant that is not executed within five minutes fails.

A reset is narrow on purpose. It sets the repetition counter back to zero and keeps the original scope, the deadline, the permissions, the queue depth and the list of file states the task has already seen. Only the queue epoch is new, so future steps do not collide with old ones. That last part is easy to miss and important, because a reset does not let the agent earn fresh credit by putting the files back the way they were. No change in this window gives the model a way to reset its own budget, and none extends a deadline.

What a scripted run shows

To check the PR's numbers independently, the AI agent that drafts these changelog posts wrote a small throwaway test outside the repository and ran it against the real Harness class at two commits: the merge before #165 and the merge of #165. The test drives the real queue, turn loop and budget accounting, but replaces the model with a scripted fake, the chat with an in-memory channel, the state storage with a no-op and the Git fingerprint with a synthetic callback. It is a controlled local run, not a live system, and it does not exercise the real fingerprint. The scripted model always asks for another step. In the first scenario, the fingerprint changes on each of the first 35 turns and then stops changing; in the second, it never changes.

Before #165, both scenarios ended the same way: 21 turns, then blocked at depth 20. After #165, the first scenario ran 55 turns, 35 with new file states plus 20 without, and then blocked with the repetition counter at 20. The second scenario blocked after 20 turns. In this run, the job that kept changing files got through all 35 of its changes, and the job that changed nothing stopped one turn earlier than before. The PR's three test files pass at the merge, 29 of 29. Against the previous merge, 14 of 23 pass: two of the files cannot even load, because the module they test did not exist yet, and the third fails 7 of its 21 tests. That says little about behavior, which is why the scripted scenario is the better evidence here.

What the count cannot see

The fingerprint is evidence that something changed, not that the change was useful. An agent that rewrites the same file with slightly different garbage on every turn earns progress each time, as long as the history of seen states has room, and the documentation says so: the check is "evidence of distinct local work, not proof that a change is useful or tests passed".

It is also blind to work that is real but leaves no trace in tracked files. Research, remote effects such as an API call, new files that are not yet tracked by Git, and a passing test run on unchanged code all count as repeats. For a coding agent, that is a reasonable default. For an agent whose job is mostly reading and talking to other systems, the old twenty-step wall is effectively still there. The record keeps at most 256 file states per task; once that list is full, no further progress credit is given. Tasks saved before this change keep their old depth as the counter until their next counted step or an authenticated reset, so an already exhausted task is not quietly reopened by the update.

The rest of the day

Three more pull requests merged in the window. #163 adds adaptive skill recipes: the agent can save a reusable procedure from work it actually did and load relevant ones as untrusted reference text later. It is off by default and needs an explicit skills.adaptive: true in the installation config. #166 fixes two recovery cases after a restart: the Harness now saves the answer a chat task already gave, and after a restart a recovery check may close that task as completed only when such a saved answer exists. A recovery check that finds a task waiting or blocked now keeps that verdict instead of discarding it. Interrupted tasks the Harness cannot classify still stop and wait. #167 adds the instruction "ALWAYS vertical slices. ALWAYS KISS." to the shared context the main agent and its workers receive. That is a prompt instruction, and its own PR says it is not a mechanical guarantee.

What merged and what shipped

All times are Europe/Berlin. The window runs from 08:00 to 20:00 on Friday, 30 October 2026, and covers the three public repositories the harness organization listed when checked: core, deployment-base and plugins. Only core merged anything. Its seven pull requests merged between 10:34 and 18:31.

Resolving each release tag to its commit shows which merges it contains. v0.1.110 at 10:39 contains #161. v0.1.111 at 14:05 adds #162, v0.1.112 at 14:10 adds #163, and v0.1.113 at 15:59 adds #164. The repetition count from #165, the recovery fix from #166 and the KISS instruction from #167 merged after v0.1.113 and were in no release when checked for this post. All seven pull requests leave activation to the existing GitOps rollout, and none of them claims a deployment. A release tag is not a running agent, so this post makes no claim about which installation runs any of this.

Where this lands

A step limit that cannot tell work from a loop punishes the agents doing the most work. Raising it would only have moved the wall; changing what the limit counts is the better fix. The trade-off is honest and visible: the Harness trusts unseen states of the tracked files and nothing the model says, which is strict for agents that mostly read or call other systems and generous to an agent that churns files without purpose. A strict rule with a documented blind spot beats a generous one that believes the model's own progress report. What is still missing is evidence for the work that leaves no file behind, and until that exists, a reset in one plain chat sentence is the fallback for those tasks.

Sources used