KarstenSamaschke
Zurück zu allen Artikeln

Notizen aus der Praxis

Not at Any Price

I count myself among the local AI folks. When a post with close to 500 likes says we will buy a 1.5TB machine at any price, it does not speak for me.

Karsten Samaschke · · 10 Min. Lesezeit

Not at Any Price

I count myself among the people who run open-weights models locally, because I care about privacy and want my data to stay in an environment I control, so whenever somebody on X writes about "local AI folks like us", I am pretty much the person being addressed. That is exactly why a post by Jun Song (@jun_song) from October 28 has been bothering me for two days now, since it claims to speak for that group and ends with a sentence that does not speak for me at all:

If Apple drops a new Mac Ultra (formerly Mac Pro) with 1.5TB of unified RAM, it will go head to head with the NVIDIA DGX Station.

Technically, they can build it right now just by linking two M5 Ultra chips together.

That gives us around 2.5TB/s of memory bandwidth.

Local AI folks like us will buy this at any price.

Screenshot of Jun Song's X post from October 28, 2026, ending with "Local AI folks like us will buy this at any price."

I replied to it on X the next day, but a reply has too little room for what actually gets to me here, and to be honest it is more than annoyance, because I find this attitude pretty disgusting when it comes wrapped in the language of our own community. My problem is not with Jun Song as a person, and not with the speculation about two linked M5 Ultra chips either, which may or may not ever ship. My problem is with the last sentence, which close to 500 people liked, and with what it says about where parts of the local AI scene are currently heading, since it has nothing to do with open weights or privacy and a lot to do with status.

Cover text reads "Local AI should run on hardware people own". Below it, a tall black server rack with rows of small light-blue status lights stands on a gray floor next to a low wooden table holding one compact dark-gray computer.

Who Would Actually Need This Machine

There is exactly one scenario where I think a machine like that is justifiable, and that is a company one, with dozens of people working on that one box all day, sharing its capacity, and with its cost spread over a whole team and a real business case. I have no issue with a company buying it, but that is clearly not what "local AI folks like us" implies, since the way I read it, the post talks about individuals, about hobbyists and prosumers, and for a single person nothing justifies that kind of money, at least not in my book.

Even with that budget, the machine would not be my pick for the work I do. Agents keep feeding huge contexts back into the model, which makes Prompt Processing far more important than raw memory size, and Nvidia is currently clearly ahead there, so with that money I would rather take something like four DGX Sparks than one giant Mac. But the point is: nobody in the open-weights crowd should have to think about this choice at all, because the whole idea was that local AI would not need datacenter money in the first place.

766 Tokens per Second on Four RTX Pro 6000

One day after Jun Song, 0xSero (@0xSero) posted a benchmark that makes the same problem even more concrete:

766 tok/s on deepseek-v4.1-flash-1017 concurrency 1 on 4x rtx pro 6000

These cards are going to the moon..

Credit: kacper-daftcode

Screenshot of 0xSero's X post from October 29, 2026, reporting 766 tok/s on deepseek-v4.1-flash-1017 at concurrency 1 on 4x RTX PRO 6000 and saying the cards are going to the moon, with credit to kacper-daftcode. It quotes his October 26 benchmark post for DeepSeek-V4.1 on the same four cards.

Granted, 766 tokens per second at concurrency 1 is an impressive number, and the recipe behind it (credited to kacper-daftcode) is real engineering work, so I do not want to talk that down. But anyone can do the math on the hardware, and by my own estimate four RTX Pro 6000 cards come to at least 40,000 to 50,000 US dollars for the GPUs alone, before anybody has bought the machine around them. Current US listings are even higher than that, since fal.ai puts a single card at about 15,000 to 16,000 USD, which makes four of them closer to 60,000 to 64,000 USD. Nobody with a normal life can afford that, and "going to the moon" celebrates exactly this price level as the place where local AI is supposed to happen.

To me this is counterproductive for the whole cause of open builds, self-hosting and Homelabs, and it sends a signal that is basically a gift to the frontier labs. When the poster child for local inference is a GPU stack in that price range, the message that lands with everyone else is that nobody can afford local AI now or in the foreseeable future, so they might as well keep paying for tokens.

We Already Know the Two-Class Version

What bothers me most goes beyond these two posts, because for me this is the entry into a multi-class society in local AI, and we already know what that looks like on the frontier side. Whoever can afford it currently burns thousands of dollars a month on tokens, maybe even tens of thousands, just to work with Claude Fable 5.1 and GPT-6 Astra, while people without rich parents, founders whose startup needs every dollar somewhere else first, and ordinary people with an ordinary income simply cannot. They get the throttled plan, the smaller model or nothing, and that gap decides who gets to build things, how fast, and with which quality.

Local open weights were supposed to be the way out of that divide, at least the way I always understood the promise: a capable model on hardware a normal person can actually buy, with no dependence on someone else's pricing, someone else's terms or someone else's logs. That is the democratic part of open weights, and it is the privacy part as well, since privacy that only the wealthy can afford is not much of a right anymore.

When the loudest voices in our own community present a 1.5TB machine bought at any price, or a stack of four RTX Pro 6000 cards, as the real local AI setup, they bring that same divide into the one place that was supposed to be free of it. Then there are the people with a datacenter in their living room who post their screenshots, and there is everyone else, who is told, implicitly or explicitly, that what they can afford is not the real thing. I get the fascination with good hardware (I run a Homelab myself, so I am certainly not innocent here), but turning the demand for ever more of it into the identity of a whole movement has nothing to do with open weights anymore, because it is about owning the biggest box and being seen owning it.

Hundreds of Likes Are a Demand Signal

These posts have consequences beyond the attitude, because vendors read them as well. Every post with hundreds of likes saying "we will buy this at any price" tells them there is a market for ever bigger and more expensive machines, which fits perfectly with their own appetite for margin, and I am not naive about how happily they take that signal. Nobody at a hardware vendor needs to be convinced to sell a more expensive product, and with posts like these we basically volunteer as the justification.

The timing makes it worse, since memory is currently one of the scarcest components in the industry. TrendForce expects conventional DRAM contract prices to rise another 10 to 15 percent and NAND Flash contract prices another 15 to 20 percent in the fourth quarter of 2026, quarter over quarter, and Micron CEO Sanjay Mehrotra expects memory and storage supply-demand conditions to be "much tighter in calendar 2027 and 2028 than they were in 2026". Asking for 1.5TB machines for individuals in a market like that pours fuel into a fire that already makes RAM, CPUs and storage more expensive for everybody, including the people who just want to upgrade one machine so a decent model runs at home.

What We Should Ask For Instead

If the open-weights community wants to demand something, and I think it absolutely should, then the demand should go the other way: models that are built or distilled for prosumer infrastructure and prosumer hardware. In practical terms that means a memory footprint of at most 256 GB, and better 128 GB, at sensible quantization, so a model fits on one or two unified-memory machines or a small stack of devices that a normal person or a small team can actually pay for.

This is not theoretical for me, since a lot of my own work already runs on models in that range. I use GLM-5.3-Flash for development tasks with my self-written agent, and I use Qwen intensively and am really looking forward to Qwen4, whether it ends up as a Flash or a dense model, because both would be fine for me as long as they stay in a size class that runs on hardware people own. Those models are where local AI actually happens for most of us, and they deserve far more attention than the next 1.5TB wish list.

Which leaves me with a question for DeepSeek and Z.ai, whose DeepSeek and GLM models currently carry a lot of the open-weights work: are you planning slimmer "Air" variants next to your Pro and Flash models, built to run well on one or two unified-memory machines? I do not know their roadmaps and I am not claiming anything about them, but this is the question I would like to hear our community ask loudly, instead of telling vendors that we will buy anything at any price.

So, please: when somebody says that "we" will pay any price, do not let that stand for you if it does not, and spend your attention and your benchmarks on the models that run on hardware people actually own. For my part, I will keep running what fits on a machine a normal person can buy, and I will keep asking the labs for more of it.

Sources used