Running a 125B model on used parts during the RAMapocalypse

The chat assistant on this site now runs on a model with roughly 125 billion parameters. It answers in two or three seconds, it runs on hardware we bought second-hand, and none of the conversations leave our network.

A year ago that sentence would have needed a rack of datacenter GPUs. This post is about the parts we actually used, the memory we had to scrounge to make it work, and the open-source runtime that ties it together.

The RAMapocalypse

Memory got expensive. AI datacenters are buying up DRAM production, and the knock-on effect reached ordinary desktop RAM: kits that used to be cheap commodities now cost far more, and large matched kits are the hardest to justify.

That matters for local AI more than for almost anything else. A big model needs somewhere to live, and once it doesn’t fit in the graphics card’s memory, the next stop is system RAM. Capacity is the whole game.

So we did what a lot of people are doing: we bought used, and we bought whatever was affordable rather than whatever matched.

The parts

That’s 96 GB of RAM for about $510, from two different brands in two different stick sizes.

Making mismatched RAM behave

Mixing memory kits works, but it isn’t free. Every one of our four sticks is dual-rank, so the memory controller is driving eight ranks across two channels. That’s a heavy electrical load, and neither kit’s rated profile (XMP) is designed for running alongside the other.

The result: the full 96 GB doesn’t run at the rated DDR4-3200. It runs at DDR4-3000 with CL16-18-18-42 timings, set by hand.

We considered the alternative. Pulling the two 16 GB sticks would leave a matched 64 GB kit that would almost certainly run its rated 3200 with XMP on. But the gap between 3000 and 3200 is under 7% of theoretical bandwidth (48 GB/s versus 51.2 GB/s), while going from 64 GB to 96 GB is 50% more room for the model. For large models, running out of RAM and spilling onto the SSD costs far more than a few percent of bandwidth.

We kept the 96 GB.

The model: 125B parameters, about 7B at a time

The model is Qwen3.8-Flash-Next, a mixture-of-experts model. We counted its weights straight from the files rather than trusting a label:

• About 125.7 billion parameters in the network itself.
• Of those, 120.8 billion live in 512 “experts” per layer, across 48 layers.
• For each token, the model routes through only 10 of the 512 experts. Together with the shared layers, that’s roughly 7 billion parameters doing the work for any given word.
• On top of that sits a 51-billion-entry per-layer embedding table, which is looked up, not computed, so it stays out of the hot path.

That’s the trick that makes this possible on a desktop. The model knows as much as a 125B model, but each step only touches a small slice of it. The challenge becomes keeping the right slice in the fastest memory.

We run the IQ2_XS quantization, which stores those weights at roughly two bits each: a 68 GB download, about 36 GB of RAM while running.

Strata does the juggling

Strata is an open-source runtime built for exactly this setup: one consumer GPU plus system RAM. Setup detected our card, compiled its engine against the ROCm we already had installed, downloaded the model and wrote a start script. On a gigabit connection the whole thing took about 17 minutes, most of it the download.

When the model starts, Strata:

• Loads about 33 GB of experts into system RAM.
• Fills about 18 GB of the 7900 XTX’s memory with the 13,545 experts it predicts will be used most.
• Keeps the conversation’s working memory (the KV cache) in RAM too, which frees more of the GPU for experts.
• Splits the experts that miss the GPU across 15 CPU worker threads, using the 13900KF’s performance cores and half of its efficiency cores.
• Drafts a few tokens ahead with a small prediction layer and lets the full model check them, so it can accept several tokens per step.

The expert cache is the part that makes it fast. In our logs, 96 to 98 percent of the experts each token needs are already on the GPU. Only 1-2% have to come across the PCIe bus.

How fast it is

• 55-68 tokens per second for normal answers. That’s faster than most people read.
• About 30 seconds from cold start to ready.
• Two to three seconds for a typical chat reply on this site.

We also gave it a short set of hard, checkable problems: two number-theory questions, a probability puzzle, a four-person knights-and-knaves logic puzzle, a counting problem, a coding task we ran against tests, and a deliberately misleading variant of the Monty Hall problem. It got all seven right. The hardest two took three to four minutes each, with the model working through about 15,000 tokens of reasoning before answering, where a frontier cloud model gets there in seconds. Seven questions isn’t a benchmark, but it’s a useful sanity check that a 2-bit, 125B model on used parts still reasons carefully.

What it runs

The assistant in the corner of this site now talks to Strata over our private network, with an API key, instead of to the smaller model we ran before. The same server also backs our internal chat interface. When nothing is asking it anything, the GPU idles, and rocm-powerd keeps the power draw down.

What we’d tell someone doing the same

• Buy capacity before speed. For big mixture-of-experts models, 96 GB at DDR4-3000 beats 64 GB at DDR4-3200.
• Mismatched RAM is fine if you tune it by hand. Expect to give up some speed and to test stability properly.
• A used 24 GB card is still the sweet spot. The more experts fit on the GPU, the fewer have to cross the PCIe bus.
• Let the runtime do the placement. Strata’s expert cache is the difference between a model that technically runs and one that’s pleasant to use.

None of this needed new hardware. It needed a model built to touch only a slice of itself at a time, and a runtime that puts the right slice in the right place.