KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

A Bigger Model Is Not a Safety Gate

This morning, a pull request for my Hayate Agent's new step planner first told it to send routine credential rotation to the top model tier because it touches production. Fifteen minutes later that rule was replaced: pick the model by how hard the reasoning is, and leave safety to the gates that apply at every tier.

Karsten Samaschke · · 11 Min. Lesezeit

A Bigger Model Is Not a Safety Gate

鍵替えに
重いモデルは
いらぬ朝

To change a key,
the heavy model
is not needed this morning.

My Hayate Agent (formerly the VanillaCore Agent) has a new, opt-in way to cut a delegated task into steps and hand each step to the least capable model tier that is good enough for it. This morning, the first version of the rule that defines "good enough" pointed the wrong way. Commit 216e393, dated 07:18 in pull request #264, told the planner to pick the tier "by operational risk and consequence of a mistake, not by how much thought it seems to need", and it put security, credentials, production and irreversible effects into the top tier. Under that rule, a routine, authorized rotation of a production credential, done from a known runbook, is steered to the top tier, although nothing in it needs hard thinking. In the author's own earlier end-to-end run, the top tier was Opus, with Sonnet for medium and Haiku for simple. Fifteen minutes later a second commit in the same pull request replaced that rule, and #264 merged into main at 07:46 with the replacement in force. Its own description calls the first rule "contrary to the requirement".

I think the correction is right, and the reason is bigger than cost. A model tier is a statement about how hard the reasoning is. It grants no permission. If a stronger model is the thing that stands between an agent and production, the safety story comes down to hoping the expensive one is careful. What constrains a production step in this harness are the gates: authorization and approvals routed back to the trusted conversation. Since 22:02 last night, the source also has an independent check of each step's evidence before the step counts as complete and the next one may start. That check cannot undo what a worker already did, but it stops a wrong claim of success from carrying the task forward. These gates apply at every tier, and the merged rule says so in the prompt itself.

Hayate Agent cover: small slate 疾風 logo and "Hayate Agent" at the top, the red two-line Japanese headline より大きなモデルは / 安全ゲートではない (a bigger model is not a safety gate), the white English headline "A Bigger Model Is Not a Safety Gate" and github.com/hayate-agent at the bottom, on an indigo-black field with a faint wave pattern along the lower edge

All times are Europe/Berlin. The window is 2026-11-04 20:00 to 2026-11-05 08:00.

What the step planner does

Pull request #254, merged at 20:52, added sequential as a second built-in dispatcher strategy next to direct. direct stays the default and runs one ordinary worker turn per task. sequential first asks the configured dispatcher model, in one turn without tools, for a closed JSON plan of one to six steps. Each step names a tier (simple, medium or high), a reason and the evidence it has to prove. A single simple step is a valid plan, and malformed output blocks the task visibly instead of being repaired.

The plan and the step index are saved before any worker turn starts, so a restart resumes from the saved plan and never plans again. Each tier has to be configured exactly: a missing or unavailable model blocks the task with a visible reason, and there is no silent fallback to the default model. A step can move up one tier only when it reports itself inadequate without having called a single tool, because only then can it not have changed anything yet. Technical failures never trigger a bigger model.

That design already treats the model choice as a cost and capability question. The rule that went into the prompt this morning quietly turned it into a safety question.

Two rules in one pull request

Both versions are short enough to compare word for word. The left column is commit 216e393 (07:18), the right column is commit f352fcb (07:34), which is what merged.

Planner prompt First rule, 216e393 Merged rule, f352fcb
Choose the tier by "operational risk and consequence of a mistake, not by how much thought it seems to need" "the reasoning and task complexity"
high means "genuinely hard or risky: unclear root cause, concurrency/data loss/recovery, security or credentials, production or irreversible effects, cross-system design" "genuinely complex reasoning beyond medium: ambiguous root-cause investigation, difficult concurrency/recovery reasoning or architecture"
Production and credentials part of high "alone do NOT require high"
Gates not mentioned "All existing authorization, safety and verification gates still apply at every role; choosing a role grants no permission and bypasses no gate."

The merged rule also gives this case as its example: "Routine authorized credential rotation with a known runbook and production read-only checks can be simple." Executing a known runbook is simple. A bounded fix in known code, with its tests, is medium. Hard root-cause work stays high.

The part I find most telling is in the evidence. The pull request's earlier probe runs expected high for credential rotation, and the description now says that expectation "was wrong and is superseded". The test cases had absorbed the same mistake as the rule, so a green run proved the wrong thing. That is why the old results are kept as history and not counted as evidence for the new prompt.

For the new prompt, the author reports two fresh runs of the real HTTP planner on homelab/claude-haiku-5-5 with medium thinking, six fixed cases each: read-only runbooks on staging and production, known credential rotations on staging and production, a bounded code fix with a test, and an unknown concurrency and data-loss root cause with a fix. All 12 classifications picked the expected tier. In 11 of 12 the plan also had the expected single step; in the second run the planner split the complex case into two high steps. The author calls this "model judgement, not guaranteed perfect plan stability", and so do I. No tools ran and no credentials were rotated in those probes.

The check that has to hold at every tier

If the cheap tier gets production work, something other than model size has to catch a bad step. Pull request #255, merged at 22:02, supplies that piece. A worker ending its reply with STEP-STATUS: complete no longer completes the step on its own word. The host first sends the original task, the step and its required evidence to the configured reviewer model and asks for an independent verdict. Only a completed verdict lets the host save the step as complete and start the next one. A missing, failed or inconclusive check blocks the task. It never replans and never upgrades the model.

The same pull request lets runtimes that support it plan with exactly one tool-free HTTP request on the dispatcher model, with no retries and no fallback. If that model cannot be served this way, planning blocks.

Two limits belong next to this. The reviewer is also a model, so its verdict is an independent judgement, not a proof. And the dispatcher documentation lists what is still missing: parallel steps, replanning after a failure, operator approval of the plan per step, and plan inspection for plugins beyond /dispatch status.

Pull request #256, merged two minutes later at 22:04, removes an obstacle to switching on any of this. Before it, an agent that switched from direct to sequential blocked every retained ready or running direct task with "Strategy direct@1 unavailable; restore the matching plugin, then resolve and resume." Switching back stranded the sequential tasks the same way. Both built-in strategies now stay registered for recovery, and new work uses only the configured one.

What a scripted run shows

The AI agent that drafts these changelog posts ran each merge's added and changed test files against the code directly before and after that merge, with Node 22 on a local machine. These are controlled tests with fake runtimes and temporary state, not a live system.

Change and tests Before the merge After the merge
#254 sequential strategy, 14 tests file does not load 14 pass
#255 HTTP planner and step check, 26 tests neither file loads 26 pass
#256 strategy switch, 10 tests 9 pass, 1 fails 10 pass
#259 skill search, 13 tests 9 pass, 4 fail 13 pass
#260 skill frontmatter, 12 tests 7 pass, 5 fail 12 pass
#262 session binding, 3 tests 2 pass, 1 fails 3 pass
#264 tier rule, 21 tests 20 pass, 1 fails 21 pass

"Does not load" means the new test imports code that did not exist before the merge. For #264, the single new test checks the wording of the planner prompt, including that the risk-based phrases are gone. It proves what the prompt says. It cannot prove what a model does with it, which is why the author's probe runs above matter. At the last merge of the window, 1f8eac5, the six distinct test files together pass 65 of 65.

The authors report broader runs: a full serial suite of 1,449 tests passing for #254, 1,450 of 1,451 for #256 with one load-dependent timeout that passes in isolation, and 37 targeted tests for #264. Those are their reports; the run described here covered only the files in the table.

The rest of the night

Three smaller fixes landed between 05:28 and 06:10. #259 trims spaces, tabs and newlines from skill search queries, and a query of only whitespace now matches nothing instead of the whole catalog. #260 accepts only a complete --- line, optionally followed by spaces or tabs, as the end of a skill's frontmatter, so ---suffix or ---- no longer close it. #262 makes the session store reject a stored binding that points to another conversation's session, so a corrupted binding can no longer show a foreign title or history.

In the plugins repository, #48 (22:34) adds a CI check that builds and tests the plugins against current core main for the exact pushed revision. The author reports one known timing failure in the core suite that passes in isolation. The deployment repository had no merges.

What merged and what was tagged

When the window closed, core main was at 1f8eac5. Each release tag sits on a merge commit: v0.1.180 on #254, v0.1.181 on #256 and, by ancestry, also contains #255, v0.1.182 on #260 and also contains #259, v0.1.183 on #262 and v0.1.184 on #264. Commit 216e393 with the first rule is not contained in v0.1.183, and v0.1.184 contains it only together with its replacement. Pull request #263 merged at 08:01, after the window, and is not part of this post.

A published tag is not a deployment. sequential is opt-in: it needs an enabled dispatcher with dispatcher.strategy: sequential and enabled, configured model roles, including the dispatcher and reviewer roles. This post makes no claim that any running agent uses it.

Verdict

Pick the model by how hard the thinking is, and pick the gates by how bad the mistake would be. A rule that mixes the two sends risk alone to the top tier and calls it caution, while the safeguards stay the same at every tier anyway. The merged rule keeps them apart and says so in the prompt. How reliably a model follows that sentence is still measured in 12 probe classifications, not guaranteed.

Sources used