interactive demo · companion to the tutorial

MoE Routing Simulator

Watch tokens travel through a sparse mixture of experts, layer by layer. Adjust the knobs and see what changes.

Experts (N) 6

Top-k 2

Layers (L) 3

Capacity 1.50

Speed (ms) 180

Load balancing on

Observation

tokens routed

of 0

active params

—

of total

FLOPs saved

—

vs. dense FFN

dropped

capacity overflow

Try these experiments

Run math and then code — note which experts they share. Then run creative and see how it routes elsewhere.
Toggle load balancing off and re-run a few times. Watch one expert capture nearly all the traffic — the "rich-get-richer" collapse.
Set top-k from 1 to 4. The active-params and FLOPs numbers update live.
Lower the capacity factor below 1.25. Tokens whose preferred expert is full get dropped — see the dropped counter rise.
Watch the trace panel: each step shows which experts the router picked for each token, layer by layer.