How Kimi K3 Runs on One CPU With 8 GB of RAM

Kimi K3, Moonshot AI's 2.78 trillion parameter Mixture-of-Experts model, can run on a single CPU with a peak of 8.2 GB of RAM. Almost none of the model needs to be in memory at once: a dependency-free C99 engine called kimi-k3-in-c reads the model's routed experts off NVMe on demand, multiplies them straight out of their packed 4-bit form, and streams the dense layers from disk one at a time. At the smallest memory preset, it takes about 33 seconds per token, but the output is byte-identical to what you get with 128 GB of RAM.

K3 had already been deployed across 32 H100s before the author, posting as /u/FareedKhan557 on r/LocalLLaMA, wrote an inference engine from scratch to run it on a personal machine: six C files, libm and OpenMP, a 176 KB binary. It has no BLAS, framework, or GPU path, and the four GPUs in his test box sat idle the whole run. His own description of the approach is "nothing clever going on," but the core observation generalizes to every large MoE model shipping right now.

Why a 2.78 trillion parameter model fits in 8 GB

Routing sparsity makes this possible. K3 has 896 routed experts, and only 16 of them fire per token. By the author's accounting, 93% of the 1.56 TB checkpoint is routed experts, which means the overwhelming majority of the weights are cold at any given moment. A conventional inference stack still wants all of them resident, in VRAM or RAM, because it assumes weight access must be fast. This engine drops that assumption. The experts never become resident. When the router picks 16 experts for a token, the engine reads those weights off NVMe and uses them once.

Two further decisions keep the memory floor low. First, the experts are stored 4-bit pre-quantized, and the engine multiplies directly against that packed form with no dequantization step, so there is no inflated intermediate copy sitting in RAM. Second, the dense trunk, the non-expert layers every token must pass through, is repacked into a single file of roughly 109 GB where layer L sits at a known offset. The engine streams it sequentially, one layer at a time. How much of that trunk stays cached in RAM determines the presets.

At the 8 GB floor, working state remains resident: activations, the KV cache, the current layer's weights, and buffers for the expert reads in flight. Everything else lives on disk.

RAM and speed

The repo documents four presets, measured on the author's dual EPYC 7763 box with NVMe storage:

Going from 8 GB to 128 GB of RAM, a 16x increase, buys roughly a 40% speedup; ~20 seconds per token is as fast as the engine ever got. More RAM stops helping because the per-token expert reads are unavoidable: the 16 experts chosen for the next token are unpredictable draws from a pool of 896, so no cache of practical size gets a meaningful hit rate on them. The dense trunk benefits from cache, but the experts do not. NVMe bandwidth and CPU matmul throughput set the floor.

The output is byte-identical at every budget in between. Memory-tiering schemes in inference systems have historically been places where numerics drift, because caching, batching, and eviction decisions change the order of floating point operations. Here the memory dial changes when weights are fetched while computation remains the same, so an 8 GB run and a 128 GB run produce the same bytes. That determinism makes the engine useful as a reference for understanding the architecture.

How you verify it without downloading 1.56 TB

The checkpoint is 1.56 TB across 96 shards, and with the repacked trunk the project wants about 1.7 TB of free disk, so committing to a download requires some planning. The repo includes a self-contained test: make && make test builds a 13-layer model with the same tensor graph and checks it against a PyTorch reference from committed fixtures, including greedy decode and the incremental path with the KV cache and carried KDA state. It runs in about a minute with no weights and no network. This avoids finding an off-by-one in an attention mask 900 GB into a download. Hardware requirements are modest otherwise: a Linux x86-64 CPU with AVX2 and FMA, and the disk space.

Is this a practical way to run Kimi K3?

The author built it to understand the architecture by implementing it. Half a minute per token means a 500-token response takes over four hours. It is a reference implementation. GPU clusters and the practical local-versus-cloud decision most teams face use local candidates that fit comfortably on the device.

Moonshot shipped the model in mid-July 2026 with a 1M-token context window, positioning it as trailing Claude Fable 5 and GPT-5.6 Sol overall while leading other open models on coding and agentic benchmarks. The full open-weight release completed by July 27. Reaction on r/LocalLLaMA was split: enthusiasm that a 2.8T open-weight model exists, tempered by the fact that almost nobody in that community has hardware that can run it normally. An open checkpoint that cannot be loaded has limited practical use. This project responds to that gap. sqliteai/waste, the "Weight-Aware Streaming Tensor Engine," implements the same split independently, keeping the dense trunk resident and streaming routed experts from NVMe behind a bounded LRU cache. Two dependency-free C engines converged on the same design within weeks of the release.

What expert streaming means beyond this project

Sparse MoE models can treat weights as a storage-tiered asset rather than a memory-resident one. Dense models gave us the habit of equating model size with required memory, and that habit shapes hardware purchasing, quantization choices, and which open releases people bother downloading. Routing sparsity breaks the equation. When 16 of 896 experts fire per token, the resident set is a small dense trunk plus a rotating sliver of expert weights, and disk capacity and read bandwidth become the main requirements. Both are orders of magnitude cheaper than RAM or VRAM per terabyte.

On one CPU with one NVMe drive, streaming lands at tens of seconds per token regardless of how much RAM you add. Lower latency would require overlapping expert reads with compute, striping reads across drives, or predicting the router's choices early enough to prefetch. This engine does none of that. It shows that a 2.78 trillion parameter model can produce correct, deterministic output through 8.2 GB of RAM and a 176 KB binary. The six C files make K3's inference path legible.