KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

One Quick Question, and the Task Was Gone

Asking an agent a quick question in the middle of a job used to cost you the job: in the VanillaCore Agent Harness, the question replaced the saved task. Core #115 keeps the earlier task alive until it finishes, runs out of its limits or is explicitly stopped, and the old version fails the new regression tests.

Karsten Samaschke · · 10 Min. Lesezeit

One Quick Question, and the Task Was Gone

An agent is halfway through a job: it is implementing a provider lifecycle, and its saved next step is to test how the provider drains. The operator types a quick question into the same chat, "What version is the second agent running?" The agent answers it, and then it stops. The drain test never runs, because until this morning my harness treated the question as the new job and let it overwrite the old one.

That case comes from the regression test that ships with the fix, core #115 in the public VanillaCore Agent Harness, merged at 11:14 Berlin time and released in v0.1.66. I like the fix because of what it refuses to do. It does not let a model guess whether a new message means "drop the old work"; the old task stays alive until it finishes, runs out of its limits or someone stops it explicitly, and I think that is the only honest default for an agent that works in the same chat you talk to it in.

A closed dark graphite notebook lies on a navy desk with a thin muted light-blue ribbon bookmark hanging from its pages, and a small blank off-white card lies separately beside it

How the job got lost

A Harness agent keeps a checkpoint for the work it was asked to do: the original request, the next concrete step, how many follow-up steps it has already taken and when the task expires. That checkpoint is what lets the agent pick up the work again after a turn ends or after a restart. Up to v0.1.65 there was exactly one checkpoint per conversation, and the task recovery documentation said so in one sentence: "A newer original chat turn replaces the session's task checkpoint."

For a chat agent, that sentence has a large blast radius. An ordinary question from the operator is a new original chat turn, even when it has nothing to do with the work, so in the tested cases the question took over the checkpoint before the saved next step was queued, got its answer, and was marked finished. The implementation task was no longer anywhere the scheduler could see it, and nothing failed, because the agent had done exactly what it was built to do. The only symptom was that the work did not continue, and somebody had to notice that and ask again.

The old behavior is reproducible. Run against the v0.1.65 source, the new regression test fails: the test expects two prompts, the question and then the resumed drain test, and the old harness sends only one. A second test queues the question while the agent is still in the middle of its working turn, which is the more realistic timing, and fails the same way with two prompts instead of three. At the window-end commit 7fc1036 both pass.

What changed in #115

Instead of one slot, the fix keeps a list of retained tasks per conversation. A new message gets its own task entry, and the earlier mandates stay in the list. When the queue is empty, or on the next scheduler tick, the harness picks up a retained task that has a concrete next step and saves it to the durable inbox before it runs. The question is answered first, then the original work continues, and the documentation now states the rule directly: "Completion of the question cannot complete another task."

Two smaller changes close the gaps around it. A turn that ends without any continuation decision used to turn into "waiting", and now it stays active for one independent check against the original request; an uncertain or failed check does not authorize anything. Results from background workers are matched to the task that started them, not to whatever question arrived last.

What I care about most is what #115 deliberately leaves out. There is no natural-language guess that a new request cancels an old one, so if I really want the old work gone, I use /stop, which revokes every retained task in the conversation. If I only want it to hold still, I pause the harness, which stops execution without throwing the work away. A smarter-sounding design would let the model decide that "never mind, do X instead" ends the previous job. That design also lets the model decide it when nobody meant it, and then you are back to work that silently disappears, only now with a plausible explanation attached.

An illustrative walkthrough

The following sequence is constructed from the regression tests and the documentation; it is not a log from a live system.

  1. The agent is working on "Implement provider lifecycle" and has saved "Test provider drain" as its next step.
  2. The operator asks which version the second agent is running.
  3. With v0.1.65, the question replaces the checkpoint, the agent answers, and the session goes quiet.
  4. With v0.1.66 and later, the question gets its own entry. The agent answers it, then the harness queues "Test provider drain" for the original task and runs it.
  5. If the operator types /stop instead, both tasks are stopped and nothing resumes.

Two timelines compare one chat on v0.1.65 and on v0.1.66: on the old version the question replaces the saved task and the work ends after the answer, on the new version the question is answered first and the saved drain test runs afterwards

What the tests check is the second timeline. They do not show how a real model behaves in a long conversation, only that the harness no longer throws the task away before the model gets a chance.

The follow-up in #126

The next fix in the same area landed that afternoon: core #126, merged at 14:53 and released in v0.1.76, deals with a parent task and its child. A pending child execution could keep the harness from assessing the parent at all. Now a pending child only blocks the execution of its own task, and other retained tasks that have no next step get their own assessment against their own original request. The same change touches the timed waits from the morning post: a scheduled recheck no longer suppresses the independent assessment, and the assessment does not overwrite the timer.

On top of that, the assessment gets more context. The harness now keeps the last eight user and assistant messages, cut to 2000 characters each, plus summaries of up to eight retained tasks, and keeps them across a restart. The test for this uses an operator correction as its example: "A merged child PR does not complete the parent task." Before #126 the assessment received no excerpt of the conversation, so a correction like that, given a few messages earlier, could not reach it. The documentation is explicit that this context is evidence and not authority: it cannot create a new task and it cannot revive one that was completed or stopped. The paired plugin change, harness/plugins #17, adjusts the assessor prompt to use it.

What this does not solve

Retained tasks are not an exactly-once guarantee. The #115 description says that resumed steps still have to reconcile side effects that may already have happened, and the documentation requires recovery prompts to check the actual state before retrying. The 20-step chain limit, task deadlines, pause and stop all still apply.

In #126 the assessor is still a model, and #127, the test-only follow-up, says plainly that its tests do not prove semantic classification is infallible. Eight messages of 2000 characters are a bounded excerpt and not the full transcript, so an important correction from an hour ago can still fall outside the window.

The core changes are merged and released; the paired plugins change is merged, and harness/plugins published no release in the window. None of that says which version any particular installation runs. My own check was limited to running the seven new task-resumption tests against v0.1.65, where all seven fail, and the same tests plus the two completion-context tests against the window-end commit, where all nine pass; the full-suite, typecheck and build results in the pull requests are the authors' reports. If you run Hermes Agent, Claude Code, Codex or OpenClaw, it is worth asking the same question of your own setup: what happens to the running job when you ask a question in between? I have not tested how those tools answer it, and nothing here is a claim about them.

Where I land

An agent that you can talk to while it works has to treat a question as a question. Before this change my harness treated it as a replacement order, and the cost showed up as silence rather than as an error, which is the expensive kind of failure. Keeping every authorized task until it finishes, hits its limits or a human explicitly stops it is less clever than inferring intent from the conversation, and it is the version I would rather debug.

Elsewhere in the window

Core #121 (v0.1.72) fixes a related trust problem: after a stopped turn, the harness could publish an older assistant reply from the transcript as if it were the result. Final replies now come only from the current turn, and cancelled turns deliver no final prose. The same change made the then unsupported /reload fail explicitly; core #122, merged less than an hour later, turned it into an alias for the graceful admin /restart.

The afternoon went into maintenance and strategy plumbing. Core #129 adds an opt-in, read-only local attach to a running session. Core #130 to #134 add durable maintenance receipts and a local maintenance CLI, and the #134 description is careful to say it tests an orderly restart, not crash safety or exactly-once effects. Core #135 to #138 make the completion, nightly, context and compaction strategies selectable at runtime through admin commands, with the selection persisted across restarts.

What merged and what shipped

This post covers 27 October 2026 from 08:00 to 20:00 Berlin time (CET), which is 07:00 to 19:00 UTC. In that window core merged #106 to #139 and published releases v0.1.57 to v0.1.90; the last release, v0.1.90, points at the window-end commit 7fc1036. harness/plugins merged #9 to #25 without a release, and deployment-base had no merge and no release. The public harness organization listed these three repositories when I checked; that says nothing about private repositories.

Sources used