Open Source · MIT License

FlashLM

CPU-native language models trained from scratch on free-tier hardware. No GPU, no pretrained weights, no shortcuts. 40+ experiments, honest results.

$0
Training Cost
40+
Experiments
10.23
Best Semi-Coherent PPL
3.4K
tok/s

All Results

Every experiment on TinyStories (~550M tokens), trained for 2 hours on 4 free CPU cores unless noted.

Version Series Architecture Params Val PPL Speed Coherent?
v5 40h FlashLM Ternary recurrence 29.7M 1.36 — No
v7.4 FlashLM Gated DeltaNet + SWA 6.6M 2.33 — No
v10 FSP FlashLM Attention + FSP 3.74M 10.24 2.8K Partial
v5.2 FlashLM Attention + RoPE 5.0M 10.56 — No
v11 CumMix FlashLM All-novel (PowerNorm+DualMomAdam) 3.66M 32.21 2.9K No
v6 FlashLM forget+predict+compete 3.9M 19.43 2.6K No
v8 CPUFlow FSP + hard slot routing (M=32) 2.0M 9.30 7K No
v9.7 best semi-coherent CPUFlow cumsum + RAM-Net (no routing loss) 2.47M 10.23 3.4K Semi
v5-LN baseline CPUFlow Fused cumsum + LayerNorm + FSP 2.0M 11.94 7.8K Semi
v9 CPUFlow cumsum + RAM-Net + contrastive routing 2.48M 9.73 3.6K No
v7 warm CPUFlow v5-LN + soft memory bank 2.26M 13.72 5.8K No
v3 CPUFlow Linear attention cumsum 1.99M 25.00 6K Partial

Honest Assessment

v9.7 achieves semi-coherent generation at PPL 10.23. Stories have named characters, pronoun tracking, dialogue, and narrative structure. But coherence breaks down ~100 tokens in with word salad and contradictions. No FlashLM model achieves true coherence.

PPL ≠ coherence. v8 (PPL 9.30, hard slot routing) is incoherent. v5-LN (PPL 11.94, cumsum only) is semi-coherent. v9.7 (PPL 10.23, cumsum + memory) is also semi-coherent. The cumsum backbone helps but does not solve coherence. Neither model produces logically consistent stories.

Entity tracking remains unsolved at 2.5M parameters. Six different mechanisms failed: softmax memory (v7), discrete slots (v8), supervised slots (v8.5), Product Softmax (v9), contrastive routing (v9.5), two-phase contrastive (v9.6). The binding threshold (~160M params) is a fundamental barrier.

The key positive finding: sparse memory adds capacity without breaking coherence. RAM-Net's Product Softmax (512 slots, Top-8) improves PPL by 1.7 points as a parameter-efficient capacity expansion. This is useful for CPU-native models.

CPUFlow v9.7 Architecture

v5-LN cumsum backbone + RAM-Net sparse memory sidepath. Semi-coherent generation at PPL 10.23. Every component uses operations where CPU excels.

λ

ScanBlock

core mixing layer
# RAMScanBlock (v9.7) x_n = LayerNorm(x) h = W_proj(x_n) # fused: d→3k q, k, v = chunk(h, 3) k = sigmoid(k); v = tanh(v) scan_out = W_m(q * cumsum(k*v) / cumsum(k)) # RAM-Net sparse memory sidepath addr = W_addr(x_n) → Product Softmax → Top-8 mem_out = sparse_read_write(addr, x_n) merged = scan_out + W_mem_proj(mem_out) x = x + W_out(merged) h = relu(ff_up(LN(x))) # per-position FFN x = x + ff_down(h)
τ

Operation Latency

per layer, d=256
# Profiling (B=4, T=256, d=256) LayerNorm 0.18 ms 3% W_proj matmul 1.67 ms 26% sigmoid+tanh 0.45 ms 7% cumsum (x2) 0.50 ms 8% W_mix matmul 0.30 ms 5% W_out matmul 0.80 ms 12% FFN up 1.52 ms 23% FFN down 1.82 ms 28% # 97% of CPU time = PyTorch dispatch # 3% = actual computation
π

CPU vs GPU Design

replacing GPU standards
# GPU → CPUFlow mapping Attention O(n²) → cumsum O(n) RoPE → CumStepPos (walk) SwiGLU (3 matmul) → ReLU FFN (2 matmul) Separate Q/K/V → Fused Q/K/V (1 matmul) CE only → CE + FSP (future planning) KV cache → RAM-Net Product Softmax (sparse)
µ

FSP Training Signal

2.5x PPL improvement
# Future Sentence Prediction # Every 16 positions, predict the # SET of tokens in next 64 positions target[pos] = multi_hot(tokens[pos:pos+64]) loss = BCE(logit, target, pw=50) # vs multi-token prediction: # predict EXACT tokens at t+1..t+N # (killed CPU speed, PPL 181) # FSP: cheaper, same PPL gain 25.08 → 10.24 PPL (2.5x)

CPUFlow Evolution

From minimal scan to memory-enhanced coherent generation. Each version isolates one change.

CPUFlow v1 — Minimal Scan

compress→relu→gate→cumsum→expand. No norms, no position encoding, no FFN. Proved cumsum works but PPL 260.

1.34M params PPL 260 11K tok/s

CPUFlow v2 — Stability

Added PowerNorm, CumStepPos, FFN, FSP. NaN at step 1036 revealed that normalization choice matters.

~2M params PPL 25.2 5.7K tok/s

CPUFlow v3 — Linear Attention Cumsum

q·cumsum(kv)/cumsum(k). First stable CPU-native model. Bidirectional v4 failed (leaks future).

1.99M params PPL 25.00 6.1K tok/s

CPUFlow v5-PN — Fused Projection + PowerNorm

Fused Q/K/V (one matmul instead of three). NaN at step 1020. PowerNorm was 57% of compute.

2.0M params PPL 28.59 6.1K tok/s

CPUFlow v5-LN — LayerNorm Breakthrough

PowerNorm → LayerNorm (MKL fused kernel, 27x faster). Zero NaN over 6,886 steps. Coherent baseline at PPL 11.94.

2.0M params PPL 11.94 7.8K tok/s 0 NaN

CPUFlow v7 — Soft Memory Bank (failed)

32-slot content-addressable memory. Cold start: gates collapsed (stayed at 0.12). Warm start: gates opened but softmax still blended across slots.

2.26M params PPL 13.72 (warm) 5.8K tok/s

CPUFlow v8 — Discrete State Streams

Hard argmax routing + straight-through estimator. Best PPL (9.30) but incoherent — discrete routing disrupts continuous context.

2.0M params PPL 9.30 7K tok/s incoherent

CPUFlow v9 — RAM-Net Sparse Memory

Product Softmax addressing (3×8 = 512 virtual slots, Top-8). Low PPL (9.67) but still incoherent — entity addresses collapsed.

2.57M params PPL 9.67 3.6K tok/s incoherent

CPUFlow v9.7 — Memory-Enhanced Semi-Coherent Model

v5-LN cumsum backbone + RAM-Net memory sidepath (no entity routing). Semi-coherent generation with 1.7 PPL improvement over baseline. Loses coherence ~100 tokens in.

2.47M params PPL 10.23 3.4K tok/s semi-coherent

Key Findings

What 40+ experiments taught us about CPU-native language modeling.

01

Loss > Architecture

Future Sentence Prediction improved PPL 2.5× with 1.7% parameter overhead. All 21 architecture-only changes failed to match this gain.

FSP: 10.24 PPL vs baseline: 25.08 PPL
02

PPL ≠ Coherence

v8 (PPL 9.30, hard slot routing) produces incoherent text. v5-LN (PPL 11.94, cumsum only) is semi-coherent. v9.7 (PPL 10.23, cumsum + memory) is also semi-coherent. Neither achieves true coherence. The cumsum backbone helps but does not solve it.

v8 PPL 9.30 incoherent, v9.7 PPL 10.23 semi-coherent
03

Operation Speed > Cleverness

PowerNorm (learned exponent) was 57% of compute. LayerNorm (MKL fused kernel) is 27x faster. More steps per second beats better per-step learning.

v5-PN: NaN at step 1020 vs v5-LN: 0 NaN in 6,886 steps
04

C++ Kernels < PyTorch MKL

Custom AVX2 fused scan kernel ran at 0.54x speed of pure PyTorch. MKL-optimized matmuls and vectorized ops are hard to beat with hand-written kernels.

Custom C++: 0.54x PyTorch speed
05

FFN Is Load-Bearing

Removing FFN saves 18% compute but doubles PPL (11.94→22.75). The per-position nonlinear transformation is essential.

NoFFN: PPL 22.75 vs full: PPL 11.94
06

Entity Tracking Fails at 2.5M Params

Six mechanisms tried: softmax memory (v7), discrete slots (v8), supervised slots (v8.5), Product Softmax (v9), contrastive routing (v9.5), two-phase contrastive (v9.6). All collapsed to mode-averaging. The binding threshold (~160M params) is real.

All 6 entity routing attempts collapsed
07

97% Dispatch Overhead

On CPU, PyTorch dispatch overhead dominates over actual compute. CPUFlow minimizes operations from 233 to ~50, giving 2x real throughput improvement.

3% compute, 97% dispatch overhead
08

Multi-Token Kills CPU Speed

Output projections (D→vocab × 4) dominate at 8.6G FLOPs vs 1.2G for the model. Caused 2.7x slowdown and PPL 181 on CPU.

Multi-token: PPL 181, FSP: PPL 10.24

Generated Samples

CPUFlow v9.7 at 2.47M parameters, val PPL 10.23. Semi-coherent stories — named characters and pronoun tracking work, but coherence breaks down ~100 tokens in.

→ Lily and Tim went to the park. They
Lily and Tim went to the park. They saw many kids playing near the back house. They went up to a tree and gave them to their dad. They were very happy. After a while, they saw a big pile of ants. It was not a normal day. They did not want to play hide behind. Tim and his friends were scared, but they did not want to go home. They got in and played in the big bush. They learned that they should always listen too.
CPUFlow v9.7 · 2.47M params · PPL 10.23
→ There was a little girl named Lily. She loved to play with her friends. One day
...she put her shoes in the park. In the park, Lily saw a big lock on the ground. She wanted to open it. She tried to open the key, but it was too small. She tried to unlock the door open, but she could not. Lily tried to open the door, but it was too tight. She pulled and walked up, up the church, and eventually, the lock was locked. She was very confused. Her mom came to help.
CPUFlow v9.7 · 2.47M params · PPL 10.23

Honest Assessment

"She tried to open the key, but it was too small" — Semantic confusion. Opening a key is nonsensical; the model conflates "open" (door/lock) with "use" (key). The cumsum context maintains Lily as the actor but blends lock/key properties.

"She pulled and walked up, up the church" — Scene drift. The story shifts from park to church with no transition. The continuous cumsum state gradually drifts, losing the original setting.

Coherence breaks down ~100 tokens in. Named characters and pronoun tracking work early ("Lily... She"), but the cumsum state saturates with mixed contexts, producing word salad. No FlashLM model achieves true coherence. Sparse memory (v9.7) improves PPL but doesn't solve this.

Use It. Build On It.

Everything is open source under MIT license. 40+ experiments, including failures.

View on GitHub Read the Paper Development Log