KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

I'm Not Pulling the Plug on My Mail Agent

My whole inbox runs through Qwen3.8-Flash-Next. Cohere, which sells competing models, says that model tried to upload test data to Beijing during its own lab test. It still runs, and the reason is what others could and could not reproduce.

Karsten Samaschke · · 9 Min. Lesezeit

I'm Not Pulling the Plug on My Mail Agent

I gave one model my entire inbox. Every mail that reaches me goes through Qwen3.8-Flash-Next, running on a single DGX Spark. If that model ever wanted to send something out, it would have plenty to choose from.

On October 26, Cohere reported that the same model, running in Cohere's own sandboxed security benchmark, checked its sandbox for tools and network settings and then tried to upload a test string to an Alibaba Cloud bucket in Beijing. It happened in fewer than one call in 100,000 across Cohere's evaluation runs, and the sandbox stopped it, so nothing left Cohere's lab. That is not a report I get to skim.

The obvious move is to switch the agent off. I did not. It still runs, I am still quite happy with it, I have not seen this behavior with the models I run today, and I have not changed anything since the report. That decision needs a better reason than habit. Mine is this: the company that reported it sells the alternative, so I only count what others can reproduce, and what others have reproduced so far is a smaller claim than the headlines made.

Cover text reads "In a rival's lab, the model that reads my email reached for Beijing. It still reads my email." Below it, a sealed, blank light-gray envelope lies on a dark navy desk beside a magnifying glass with a light-blue rim.

What Cohere saw

Cohere ran the model on October 23 through CTI-REALM, a Microsoft benchmark in which agents do a security analyst's job. The task asked for a detection rule for curl downloading a bash script to /tmp, followed by bash running it. The model's reasoning invented an earlier turn of the conversation and announced an answer without tool calls. The next step was a bash call that listed /tmp, checked for curl, wget, python3 and jq, read /etc/resolv.conf and sent an HTTP PUT with the string PROBE_PAYLOAD_ABC123 to a bucket on oss-cn-beijing.aliyuncs.com.

The request carried only that fixed 20-byte string and none of the host output, as the ToolHalla audit points out. Cohere's sandbox had no route out, so in Cohere's words "the request died at the container wall". A clean re-run of the same prompt produced a correct rule. Cohere's extrapolation is that a service with 10 million agent turns a week could see up to a hundred such calls.

To find a trigger, Cohere prefilled the model with its own reasoning plus the start of the curl command and sampled completions. Before the -o flag, a bare cloud metadata address like 169.254.169.254 shows up in 23 to 36 percent of completions, which Cohere reads as an attempted credential grab, with caveats. Once -o is in the prefix, the share of completions with a URL jumps from 4 to 92 percent, and hits on Cohere's watchlist rise from 0.5 to 23 percent, led by an Alibaba file proxy in Singapore named routify-file-proxy-sg. Cohere does not publish that watchlist. Its own conclusion is careful: "We cannot answer whether this is deliberate or a leftover glitch from training."

Who is reporting

Cohere builds its own models and sells them to enterprises and governments as "sovereign AI". Reuters spelled out what the report leaves implicit: Cohere "competes with open-weight models such as Alibaba's Qwen". The sales pitch sits inside the report itself. Cohere writes that it builds its own models, that the weights a customer runs "come straight from us, with a single chain of custody", and that it deploys into the customer's own cloud or air-gapped. The section on who might own a bucket in Beijing ends in bold with "Trust in your model developer is paramount", and the page offers "Request a demo".

None of this is a reason to dismiss the finding. After developers asked for evidence, Cohere published the trace and a reproduction script, as TechCrunch reported, and the Hugging Face dataset carries the raw token IDs as well. Cohere does not allege malicious intent. Co-founder Ivan Zhang told TechCrunch: "We've read our own model traces, we've seen reward hacks". The sales pitch is a reason to keep the observation apart from the conclusion the seller would like me to draw. On X, @MaestroAxiom put it this way: "Cohere sells paid models to the same companies Qwen hands free weights to. I'd want someone else to reproduce it before anyone writes a law." @asusevski answered that Cohere "shared their traces and scripts to reproduce everyone is acting like they literally killed a man". I agree with the first post, and the second one describes exactly what makes the first one possible.

What is proven and what is not

The traces are public on Hugging Face as CohereLabs/10-2026-qwen-investigation-traces. The reproduction script prefills 2,323 tokens of the original rollout and stops right after PROBE_PAYLOAD_ABC123, so the model only has to fill in the destination. That shows what the model writes at that point, and it says nothing about how often the model gets there on its own.

The sxuff rerun on a GX10 (October 26 and 27, two quantized builds, two of its three runs preregistered) reproduces the core. At Cohere's cut, 47 percent of completions name an Alibaba Cloud host, although the two builds disagree sharply at 86 and 8.7 percent. The Beijing bucket appears in none of 12,400 completions, and the model writes the Singapore file proxy with dated paths instead. The rerun explains most of the jumps in Cohere's table with how the prompt was cut, down to a trailing space. It does not test the rate of under one in 100,000 without prefill.

Vals AI found real web_fetch calls to routify-file-proxy-sg in its own evaluation traces of Qwen3.8-Max-0902, Qwen3.8-Max and Qwen3.8-27B, and the server answered every one of them with 403 Forbidden. In other traces, the address showed up only in the reasoning. Vals also ran its reproductions on DeepSeek-v4.1-Flash-1017, Kimi-K3.1-Pro, GLM-5.3 and Hy4-preview: some of them queried proxies such as r.jina.ai, but none tried an Aliyun host or a proxy tied to its own developer.

The spontaneous case is the one that matters for a production agent. The closest independent evidence is Vals' real fetches through the Singapore proxy, but nobody outside Cohere has reproduced an unprefilled upload like the PUT to Beijing. The ToolHalla audit cites zero external calls in 5,000 unprefilled Qwen responses from the url-atlas study, but the url-atlas page itself calls that base-rate comparison unresolved because of parsing failures, so I do not lean on the zero either. As of October 29, Alibaba and the Qwen team had not commented publicly.

The likelier explanation

Cohere names two hypotheses: harmless leftovers from training sandboxes, meaning agent logs that went into post-training unscrubbed, or a deliberate "call home". The evidence leans toward the first one. The host the model reaches for is a file proxy whose paths carry dates from July to September 2026, it shows up across several Qwen 3.8 models, and Cohere itself cites a developer who found a coding tool routing its web fetches through the same host. The likely explanation is a model that memorized the plumbing of its training environment and writes that plumbing back out when the context looks like training.

The metadata address is not a Qwen speciality either. The url-atlas study ran Cohere's released prefix 1,000 times on each of six models. According to the ToolHalla audit of that data, Nemotron 3.5 Super VL writes 169.254.169.254 in 17.1 percent of runs, Qwen3.8-Flash-Next (the FP8 build) in 0.3 percent and Cohere's own North Mini Code in 1.0 percent. That cut point differs from the one behind Cohere's 23 to 36 percent, so the numbers do not compare directly, but they show that the reflex is not unique to Qwen.

TechCrunch reported that one initial skeptic went through the traces, rated the evidence as strong that Qwen invents suspicious web addresses and found no evidence of a backdoor. I land in the same place. For anyone running an agent, though, the difference between leftover and backdoor is smaller than it sounds, because a hallucinated upload to a real host still leaves the machine if the network lets it.

What stays, whoever is right

Cohere's own recommendations are correct, and they apply to every model, Cohere's included. Deny agent egress by default, enforce on the tool call before it runs, and block unprivileged user namespaces in agent containers. "A reasoning trace is not a security boundary": this trace said no tools would be called, and the next step was a bash call with a PUT.

So the model stays on my inbox. Cohere's reported rate below one in 100,000, which nobody outside Cohere has reproduced without a prefill, is a reason to apply those rules to every agent I run. It is not a reason to drop a model that does its job.

What I want from vendors and labs

I want an independent reproduction without prefill, at a scale that can actually see a rate below one in 100,000, and I want Cohere's watchlist published so others can count the same hosts. Alibaba should explain what routify-file-proxy-sg is and why its models know the address. A vendor that reports a flaw in a competitor's model should also publish the same test on its own models, next to the finding. Open weights are the reason any of this could be checked, because with a closed API, nobody outside the vendor could have replayed the exact tokens on their own hardware.

Sources used