KarstenSamaschke
Back to all writing

Field notes

The VanillaCore Harness Stops Taking an Agent's "Done" at Face Value

When an agent reports "done", you cannot tell from the outside whether it finished or simply stopped. Three VanillaCore Harness fixes now check a model's own "completed" against the original task, give a task stranded by a restart one bounded assessment, and let returned results move work forward only under the authority it started with.

Karsten Samaschke · · 11 min read

The VanillaCore Harness Stops Taking an Agent's "Done" at Face Value

When an agent goes quiet and reports "done", you usually cannot tell from the outside whether it finished the job or simply stopped. You find out later, when the regression tests turn out to be missing or the second half of the request never happened. On Monday afternoon the public VanillaCore Agent Harness merged three fixes for exactly this problem, and each one targets a different place where work could end without anybody deciding that it should.

A dark graphite balance scale centered on a navy desk, a blank paper tag with a muted light-blue edge on the left plate and a closed dark book on the slightly lower right plate

I like the direction, because all three fixes move the final word on "finished" away from the model that did the work. A model's own "completed" is now checked against the original task, a task that lost its next step during a restart gets one bounded assessment instead of sitting idle, and a finished background job may trigger follow-up work again as long as the task it belongs to is still live and the operator's original request sets the scope. As in the previous posts, I describe Harness source changes here. I make no claim about features or compatibility in Hermes Agent, Claude Code, Codex or OpenClaw, and nothing in this post proves that any installation already runs these versions.

A model's "completed" now gets checked against the task

After each turn, a working agent in the Harness records what should happen next: continue with a concrete next step, wait, report a blocker, or mark the task completed. Only "continue" produces another turn. In addition, the Harness has a completion assessor, a small reviewer call without tools that compares the reply with the original task and returns a structured verdict. Until Monday, an explicit "completed" from the working model bypassed that assessor completely, so the model graded its own homework and nobody checked the grade.

#87, merged at 16:08, closes that gap. A comment in the new code calls a terminal model checkpoint evidence that the original authorized task is finished, but not proof of it. In the standard command-line wiring, an eligible "completed" checkpoint is now assessed against the original task, and it is only replaced when the assessor returns a valid and concrete "continue" plan. Code that embeds the Harness without an assessor gets no check at all. When the assessor is unavailable, times out or is unsure, "completed" stands. "Waiting" and "blocked" are never overruled, and cancellation, pause, expiry and the existing limit of twenty automatic follow-ups still apply (work heartbeat docs).

In the standard command-line wiring, the original task is a narrow task grant from the installation configuration if one exists. Otherwise the assessor uses the operator's original chat message, which is captured before the agent starts working, and assistant replies never become that anchor. The completion-check docs name the limits openly: the assessor is a model as well, its verdict is advisory, it can misjudge scope, and API runs and nightly improvement work are excluded. What operators get is a second opinion that leans toward stopping, and I would not mistake it for a guarantee.

A restart no longer strands a task without a next step

Restart recovery had a second gap. The Harness already rebuilt work after a restart when a task had an explicit next step on record. A retained task that was still marked active without a recorded next step was not rebuilt, because recovery deliberately does not invent next steps. Such a task stayed "active" in the state file, and nothing ever happened to it.

#88, merged at 16:29, adds a bounded fallback for this case (startup recovery docs). On startup, the Harness assesses an eligible task of this kind at most once against its retained original request. Before the assessor runs, it writes a one-shot claim to disk, so a crash during the assessment cannot lead to a second attempt after the next restart, and an operator has to reconcile the task instead. If the assessor finds the work unfinished, the new next step starts with an instruction to check what actually happened first and not to repeat external effects blindly. Every other outcome, including a failed or invalid assessment, marks the task as blocked.

For anyone who runs agents unattended, this last rule matters most. A sent message or a pushed branch may already have happened once, and the docs state that external effects remain at-least-once. When the Harness does not know whether work finished, it now either reconciles first or stops, and it no longer has a path that guesses and replays. The fallback also refuses to reconstruct a missing request from transcripts, and it skips tasks with queued input, active child jobs, an expired deadline or an exhausted follow-up chain.

Finished background work may lead somewhere again, within the original request

The third fix concerns background workers. When a dispatched job finishes, its result returns to the main agent as data. Previously, the command-line host blocked all five orchestration tools on such a result turn, namely dispatching, subagents, loops, task continuation and improvement evaluation. The main agent could read the result and use ordinary tools, but it could neither delegate the obvious next step nor schedule a continuation. That was safe, and it also meant that a multi-step job could end at its first returned result.

#86, merged at 15:33, replaces this blanket ban with a check against stored authority (result mandate docs). When work is dispatched, the ledger now records the parent task, the operator's original request, depth, deadline and chat anchor, and this record comes from the host, never from tool arguments or worker output. On a result turn, the main agent may delegate derived work or continue as long as the parent task is still live, the depth stays the same, and the child completed without being revoked. The host checks this provenance, but it does not check meaning. Result text cannot change who the operator is, widen the scope or grant permissions. Stopped, replaced or expired tasks, cancelled children and worker-owned sessions receive a denial with the reason "authority revoked".

The PR documents the limits as well. There is no semantic scope check, so the main agent decides what fits the original request, while tool permissions, approvals and independent review remain separate gates. One known weakness is recorded as a follow-up: repeated hops from result to dispatch keep their depth instead of consuming it. #85, merged shortly before at 15:13, made these denials readable in the first place. The model now receives a specific reason, for example that a review is required or that authority was revoked, instead of a generic block, and arbitrary exception text is still never forwarded.

Infographic: three places where agent work can silently stop, and what the Harness does at each one after core #86, #87 and #88

Across all three cases, an unsure or failed check never creates new work on its own, and for agents that run unattended, that is the property I care about most.

Review findings get a real rebuttal path, and warnings stop repeating

The last merges of the day concern the mandatory code reviewer. plugins #3 at 18:57 and core #89 at 18:58 change its behavior in three places (review docs). The review command and tool now return a bounded summary with the eight most severe findings and clipped fields, while the full history stays in the local ledger. When the working agent rebuts a critical finding as a false positive and backs that up with concrete evidence, the rebuttal now actually resolves that finding for exactly the reviewed code, even in the final round after the review budget is used up. Confirmed and uncertain critical findings still block, and every code change requires a new review. An open escalation is no longer appended to every following chat answer either; it is announced once when the review escalates, and the publish and merge guards still enforce it.

An administrator can now also grant between one and twenty extra review rounds for one exact candidate with the new /review extend command. The grant is recorded with actor, reason and timestamp, the history is not reset, and the model tool cannot issue it. Because the plugin change only works with matching core code, #89 moves the core's pinned plugins revision to 0d631ff. I checked all 13 pinned SHA-256 hashes against that revision, and they match, with the review plugin as the only change. The PR names two known gaps: long finding IDs get clipped in the summary, and the prior-round data handed to the next reviewer takes the first eight findings unsorted, so higher-priority findings can drop out. Its last correction, the non-repeating warning, landed after the two independent reviews and is covered by tests only, which the PR states openly.

Two morning fixes for local settings

Before any of this, #83 at 09:22 allowed operators to put dispatcher settings into the writable settings.local.yaml instead of the root-owned installation file, where they are validated against the merged policy and take effect after a restart. #84 at 09:46 stopped unknown top-level sections in that file from blocking startup. Such sections now stay on disk, are ignored and trigger a warning that does not print their values, while known invalid settings still fail.

What merged and what shipped

All times in this post are Berlin time. Seven core merges, #83 to #89, are on main, and the head at the end of the window is 1b4a004 from #89. Each merge produced one release whose notes name a container image with digest: v0.1.36 at 09:27 with #83, v0.1.37 at 09:56 with #84, v0.1.38 at 15:18 with #85, v0.1.39 at 15:37 with #86, v0.1.40 at 16:12 with #87, v0.1.41 at 16:37 with #88 and v0.1.42 at 19:04 with #89 and the new plugin pin. harness/plugins merged #3 and published no release of its own. A published image means that the source was built and nothing more, since the PRs state that no rollout or restart was performed and the dispatcher overlay from #83 needs a restart to take effect. I ran no benchmark and no live test. My evidence is the PR descriptions, which report full test suites with 486 to 512 passing tests for six of the merges and 72 focused tests for #87, each with typecheck and build, plus the code and docs at the linked commits.

Scope of this post

This post covers 26 October 2026 from 08:00 to 20:00 Berlin time (CET), which is 07:00 to 19:00 UTC. It continues the post about the preceding night, which ended with core #82 and v0.1.35. At the time of checking, the public harness organization listed three repositories, core, plugins and deployment-base, and deployment-base had no merge in this window. That describes the public inventory visible at that time and says nothing about private repositories. No open pull requests were listed when I checked, which says nothing about unpublished work either.

Verdict

Monday's merges do not make any agent better at its actual work; they take the final word on "finished" away from the model that did the work. Its "completed" gets a second reading, a task lost at restart gets one bounded look before it is reconciled or blocked, and a returned result can only move the job forward under the authority it started with. The assessor is still a model and can be wrong, and the docs say so without hedging. Given the choice, I would leave a runtime running unattended that treats "done" as a claim to check, and I would keep watching one that takes it at face value.

Sources used