KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

The Night Shift That Grades Itself

Since v0.1.104, the VanillaCore Agent Harness runs nightly improvement experiments by default. I tested what core #155 switches on, what stays off, and why 45 "improved" entries in one night are still the agent grading its own homework. Plus #156: a receipt for every refused wait.

Karsten Samaschke · · 9 Min. Lesezeit

The Night Shift That Grades Itself

An agent that runs experiments on itself overnight sounds like the first line of an incident report. Since release v0.1.104 on Thursday morning, it is the default in the VanillaCore Agent Harness: between 23:00 and 05:00, an agent without a policy of its own can spend up to 20 turns trying out small, isolated changes and writes down what came of each one. Until then, the loop stayed silent unless an operator had explicitly switched it on. The release says nothing about which installations already run it.

I think on by default is the right call, but only if you read the default for what it is. The window, the turn budget and the report destination are enforced by the Harness. Pull requests and merges are not: with the default policy, the agent is told that it may not open one, and that instruction is the whole fence. The code change in core pull request #155 is a single word, false became true; the rest of the pull request is about proving that this word does not switch on anything else.

A dark graphite desk lamp lights an open notebook and a closed laptop on a navy desk, centered in the lower half, under the headline My agent harness now experiments every night by default. Its results are self-reported.

What the default switches on

The nightly loop already existed, but before #155 an operator had to write enabled: true into a policy file. #155 changes what happens when that file is missing or does not mention enabled at all. Both cases now count as enabled. An agent whose file only lowers the turn budget gets the loop too, with that lower budget, and an explicit enabled: false still switches it off and is left as it is.

What runs is deliberately small. The window is 23:00 to 05:00 in the Harness's own timezone, UTC if none is set, with a budget of 20 working turns. The loop only starts when no other conversation has active or queued work, and it does not start a new experiment at the morning boundary. Each turn has to return a structured receipt with a hypothesis, a baseline, a candidate, the evidence and an outcome of improved, rejected or inconclusive. A turn that only declares itself done does not count as an experiment or end the night, but it still uses up one of the 20 turns, and three turns in a row without a new receipt end the night with a failed-night diagnostic instead of burning the remaining budget on status checks. The loop still needs a nightly strategy, and the command-line Harness supplies the bundled one from the plugins repository unless an operator disabled it.

Two columns show what the new default in core #155, from v0.1.104, switches on and what stays off by default. On: the 23:00 to 05:00 window in the Harness timezone, a budget of 20 turns, a stop after 3 turns without a new experiment, and a morning report. Off by default: pull requests, merges, an evaluator and a guessed chat target. A note says that pull requests and merges are agent instructions, not a sandbox, and that an explicit enabled: false still opts out.

The morning report goes to an explicitly configured destination, then to the Harness's home address if there is one, and otherwise into a local file under improvement/reports/. No channel is chosen by guessing. Permission for a pull request needs pullRequest: true in the policy, a merge needs merge: true on top, and the Harness rejects a policy that allows merging without pull requests. What it does not do is stop an agent that ignores its instructions. The documentation says so directly: these flags "express permitted workflow, not a capability sandbox", and the loop is not meant as a security boundary or unattended release authority yet.

Who the new default actually reaches

The part of #155 I find most useful is an audit that a lot of default changes skip. The evidence file in the pull request, written by the developer agent that made the change, checks the three agents listed in the current overlay registry and states for each one what the new default would actually do.

The first agent, severus-snape, already had its own policy with the loop enabled, a Berlin night window, 200 turns and its own pull request and merge rules, so the new default changes nothing for it. The second, che-guevara, has no versioned policy in its GitOps repository, which means the default would apply after a normal image rollout, but only if its private home holds no policy either and a nightly strategy stays configured, and the audit did not read that home. For the third, randy-waterhouse, the read-only clone failed, and the file says plainly that its policy is unknown. The conclusion in the file is that no statement about actual live activation is possible, and I prefer that to a confident "all agents are now enabled".

The same file also counts an existing raw report for the night of 28 October: 54 turns and 52 logged experiments, of which 45 are marked improved, 4 rejected and 3 inconclusive. Those are the agent's own receipts, not independently verified adoptions, and the report itself labels its measurements as reported. A ratio of 45 out of 52 is exactly why that label matters.

The other merge: a receipt for a refusal

The second change of the day is smaller and addresses a different gap. When an agent registers a timed wait for something, such as a release image, the Harness wakes it up later for a recheck, which I described in The Image Shipped, and My Agent Never Noticed. During that recheck, a guard checks every tool call against the authority the waiting task still has, and it refuses the call when that authority is gone. The model only ever gets a sanitized "authority revoked", and until now the Harness kept no record of which check had failed. One earlier wait, which the documentation calls the PR72 denial, was refused this way, and afterwards the retained evidence could not show which check it was.

Core #156 does not explain that old refusal, and its description says so in its first sentence. It makes the next one explainable. When the Harness has an execution ledger, the guard now writes a small receipt into it for six reasons: the Harness is stopping or the turn is being cancelled, the input no longer owns the turn, the deadline is missing, invalid or reached, the task cannot be found, the task was stopped, or the recheck sequence no longer matches. Each receipt holds the reason, the wait sequence and the time, plus the task ID, task sequence and turn ID where they are known, and the ledger keeps the newest 100 across all tasks. Tool arguments, transcripts and exception text are not stored, the model still sees the same sanitized answer, and if writing the receipt fails, the tool call is still denied, just without a receipt.

What I checked

I copied the two test files that #155 added or changed onto the commit right before it, the code of v0.1.103, and ran them there. Of the 8 tests, 5 passed and 3 failed: the missing policy file, the file that only sets a turn budget, and the defaults test all came back disabled. The case with an explicit enabled: false passed before and after, as it should. At the #155 merge and at the end of the window, all 8 passed. For #156, I ran its three test files on the #155 merge: 3 of 12 passed, all three of them chronology checks of the old PR72 wait, and all 9 receipt tests failed. At the #156 merge, all 12 passed. These are controlled local tests with fake runtimes, not a live night or a running agent.

The rest comes from the authors. #155 reports its focused tests passing and a full suite that did not pass, 872 of 875, with the same three failures as on the baseline. It also states that there were no Core Actions runs or status checks for that head and that branch protection on main requires no approvals and no status checks. #156 reports 41 focused tests passing in an independent review, makes no claim of a full-suite, required-CI or plugin_review pass, and was merged four minutes after it was opened. Its review also names two follow-ups it did not do: validation of the receipts when they are written, and more permanent tests for malformed ledgers and edge cases.

What merged and what shipped

This post covers 29 October 2026, 08:00 to 20:00, Berlin time (CET), which is 07:00 to 19:00 UTC. Core merged two pull requests: #155 at 08:57, first released in v0.1.104 at 09:02, and #156 at 12:28, released in v0.1.105 at 12:34. The plugins and deployment-base repositories merged nothing in this window.

A release tag is not a deployment. Nothing here says which version any installation runs, and the improvement documentation states that publishing, merging and building do not update a running installation.

Where I land

An improvement loop that stays off until someone remembers to enable it produces no evidence at all, so I would rather have it on. The Harness enforces the time, the budget and where the report goes. Everything that decides whether a change is good, and whether it may leave the night session, still rests on the agent following its own instructions and grading its own experiments. The optional evaluator is there, but its verdict is not wired into any gate, so 45 improvements in one night are 45 entries in the agent's own report card. That evaluator gate is the change I want next, before anyone writes merge: true.

Sources used