Two preprints went up a day apart at the start of this month, and they reached me pre-summarized: mixture-of-experts models are full of redundant experts, the auxiliary load-balancing loss put them there, and you can therefore delete half of them at inference, recover the memory, and pay almost nothing.
I read both. Neither says that. One of them says close to the opposite, in a sentence its authors set in plain type: βA router trained with load-balancing pressure leaves little slack for pruning.β
The underlying intuition is not stupid, which is why it spreads. The load-balancing term is a uniformity pressure. Its minimum sits where every expert receives the same share of tokens and the same share of router probability mass. It was introduced for systems reasons, to stop synchronisation stalls and under-trained experts, and nothing about it rewards experts for being different from each other. If you squint, a loss that pushes toward uniform usage looks like a loss that pushes toward interchangeable parts.
Something does homogenize. It is not the experts. The measurement that shows it is four months older than either preprint, and the quantity it turns on is a training coefficient someone else picked before you ever loaded the weights.
arXiv:2609.04575, submitted 4 September 2026, is Xing Chen and Hengshuai Yaoβs βTraining-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models.β Its object of study is Qwen3.6-35B-A3B: forty MoE layers, 256 experts each, eight selected per token. The routed experts hold 32.2 billion of the modelβs 35.9 billion parameters, 89.6% of it, and each token touches 3.1% of them. That asymmetry, as the paper puts it, is what makes MoE compression βattractive and awkward at the same time.β
Then they go looking for the fat. Normalized activation entropy is 0.889 on WikiText and 0.915 on code. Never-activated experts per layer: median zero, maximum one. Keeping the top half of the experts in each layer by routing mass retains 88.8% of that mass on WikiText and 81.3% on code, and in the worst layer, 64.2%. The specialization is sharper than I expected. Mean Jensen-Shannon divergence between the two domainsβ per-layer expert distributions is 0.258 nats, against a resampling noise floor of 3.2 Γ 10β»β΅ nats, roughly 8,200 times the floor. The top-64 expert sets for the two domains overlap at 0.164, which is below the 0.250 you would get by drawing at random: the domains are not merely using different experts, they are avoiding each otherβs. The union of two domainsβ top-128 sets already covers 198 of the 256.
Which is where the sentence quoted above comes from, and why the paper spends most of its length somewhere else. Fine-grained routers renormalize the top-k router weights to sum to one, and that renormalization was implicitly calibrated to the k used in training. Cut k at inference and you have changed two things at once: which experts fire, and how loud the whole expert branch is. Chen and Yao separate them by activating the top kβ experts while normalizing against the probability mass of a larger top-kβ reference set. One integer, no training. Going from eight active experts to four costs 4.65 MMLU points under standard renormalization and 0.35 points with kβ = 16, while halving routed-expert compute. It replicates on the eleven-times-larger Qwen3.5-397B-A17B, where ten to five loses 0.55 points. Dropping renormalization altogether is, in their word, catastrophic.
That is a compute result, not a memory result, and the two are not interchangeable: pruning experts saves memory but not compute, because k is unchanged and each token still runs through the same number of expert feed-forward networks; reducing k saves compute but not memory, because every expert still has to be resident.
arXiv:2609.04453, submitted 3 September 2026 (a day earlier, not the same day, if you saw them paired), is Berkcan Kapusuzoglu and colleagues at Capital One, βWhen Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models.β They name a regime rather than a property. Over-dispersed routing is what you get from an unusually high auxiliary coefficient: Ξ»_aux = 0.9 in gpt-oss-20b, against roughly 0.001 in Mixtral-8x7B-Instruct. The paper puts it at roughly 900 times Mixtralβs default and 90 times Switch Transformerβs. It is a dial, not a fact about mixture-of-experts.
In that regime two things break, and both are worse news for the delete-half story than that summary implied.
Perplexity stops predicting accuracy. At 25% pruning, thirty-two experts per layer down to twenty-four, random deletion produces the lowest WikiText-2 perplexity of any configuration they tested β 40.95, against 126.02 for the unpruned model β and near-worst GSM8K, 78.54% against a baseline 88.86%. Throw away a quarter of the experts at random and the perplexity improves threefold while the arithmetic gets worse. On Mixtral, which is not over-dispersed, perplexity behaves itself: 4.78 unpruned, 6.00 under activation-aware pruning, 8.31 random, 14.35 under a router-uniformity criterion. The inversion is a property of the regime, not of pruning.
And no single importance signal wins. Activation-aware scoring holds competition maths at 73.33% on AIME 2025 but drops GPQA to 47.72% from a 64.47% baseline; frequency-based scoring trades the other way. The authorsβ own method targets the worst-affected domain rather than the average, and posts the smallest worst-case degradation, with GSM8K actually rising to 92.87%.
Then the sentence that did not survive into the version I was handed, and which the authors put in themselves: βwe therefore frame MESAβs contribution as domain-fair accuracy retention at a reduced memory footprint, and make no per-request speedup claim.β Removing experts leaves top-k unchanged. On gpt-oss-20b the MXFP4 expert weights run about 1.6 MB per expert per GPU, which is not the binding constraint on single-request throughput.
So one paper is a warning that a high balancing coefficient makes pruning harder to steer, and the other is a calibration fix for a different axis entirely, with a specialization measurement attached that cuts against blanket redundancy. Between them they are closer to a rebuttal of that summary than a source for it.
There is a cleaner instrument, and it predates both. arXiv:2605.12476, 12 May 2026, βRouters Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts.β Two 1B sparse MoEs trained from scratch on 50B tokens, identical in every respect except the routing rule. That is the controlled comparison the question needs and the one I could not find anywhere in the pre-2026 literature.
Off-diagonal mean cosine similarity between router weight vectors, at layers 0, 4 and 8: with the standard auxiliary load-balancing loss plus the z-loss, the recipe behind Switch, OLMoE and Mixtral, it is 0.63, 0.63 and 0.57. With loss-free bias balancing, same architecture, same tokens, it is 0.32, 0.18 and 0.13. The auxiliary loss, in their phrasing, makes distinct router directions βnearly three times more similar.β
That is the homogenization, and it is in the router. Which is a different claim with different consequences. A collapsed router chooses badly among experts that may remain perfectly distinct, and that is consistent with both September results: Chen and Yaoβs strongly specialized experts sitting underneath a well-balanced router, and Kapusuzogluβs importance scores going uninformative once the router has been pushed toward uniformity.
Consistent with, not the same as. The three papers measure three different quantities at three different scales: cosine similarity between router weight vectors in a 1B model trained from scratch, routing-mass entropy in a 35B production model, and the behaviour of a 20B model at a balancing coefficient nine hundred times the usual one. Neither September paper cites the May one. I am reading them as the same story because the mechanism lines up, not because anyone has shown that it is.
The same table carries the number that changed how I read the rest of it. MaxVio measures worst relative overload across experts, averaged over the nine layers: 0.526 for the auxiliary loss, 0.084 for loss-free bias balancing, and 0.037 for the authorsβ parameter-free K-Means router, which keeps a running centroid per expert and assigns by cosine similarity. The loss whose entire purpose is balancing load balanced it about six times worse than not having the loss at all. Perplexity told the same story: 15.09 training and 20.54 on C4-en for the auxiliary-loss model, against 15.01 and 20.40 loss-free. The K-Means router pays 2.6% training perplexity, 15.01 to 15.40, for the best balance of the four.
One 1B run is not the field, and I would want this replicated at a scale where the economics bite before treating it as settled. But it is a controlled experiment with a single variable, which is more than the September preprints were built to provide.
Eigen, Ranzato and Sutskever described the failure all of this was built to prevent thirteen years ago, four years before the soft loss existed, in Learning Factored Representations in a Deep Mixture of Experts (December 2013). Their test beds were jittered MNIST and speech monophones, with no language modelling anywhere in the paper:
SGD by itself results in a degenerate local minimum: The experts at each layer that perform best for the first few examples end up overpowering the remaining experts. This happens because the first examples increase the gating weights of these experts, which in turn causes them to be selected with high gating weights more frequently. This causes them to train more, and their gating weights to increase again, ad infinitum.
Their remedy was not a penalty term. It was a hard cap: zero an expertβs gate when its running cumulative assignment exceeds the mean by more than a margin, renormalize the survivors, and lift the constraint after an initial phase. A constraint applied and then withdrawn, rather than a gradient pressure held on for the entire run. Thirteen years later the controlled result is that a parameter-free centroid router balances better than the loss built to balance, which is close enough to the same idea coming back around.
MoEXBench, 22 August 2026, ran ten MoE models from 30B to 235B through pruning, quantization and KV-cache compression, alone and stacked. For Qwen3.6-35B-A3B the BF16 baseline is 69.38 GB at a workload-average score of 0.8262. Prune 25% of the experts and you get 52.04 GB at 0.8196. Quantize instead and you get 21.39 GB, a 69.2% reduction, at 0.8263, indistinguishable from the model you started with. Quantization compressed almost three times harder and cost nothing measurable; pruning compressed less and cost more. Their own summary line is that βexpert pruning is the dominant degradation source,β and their deployment note is that expert pruning βreduces memory but not active computation, so smaller MoE models may see little or no speedup.β
The hardware section puts numbers on that, and they break the neater story I was about to tell. On an H100 at 32K context, prune-only holds throughput at 0.94 to 1.03 times baseline: parity, not a speedup. Quantization at Q4_K_M returns 1.38 to 1.51 times. So quantization is not only the better memory play, it is the one of the two that moves throughput at all. The paper files that under quantizationβs weakness, since 1.4Γ is a poor return on 2.8Γ of compression, which is a different complaint from mine.
One 2026 result cuts the other way. arXiv:2608.07890, 8 August 2026, ranks experts by how much a router-only LoRA moves their router weights, prunes the least-moved, and reports memory down 49% and per-token latency down 37% on Mixtral-8x7B-Instruct with half the experts gone. Latency falling while k is untouched is presumably a memory-bandwidth or expert-parallel communication effect rather than an arithmetic one, and eight fat experts are a different object from 256 small ones. The same abstract carries the accuracy, which the headline does not: MMLU-Pro goes from 44.83% to 27.54% with half the experts removed. The paperβs βnearly 1.8 timesβ is measured against random and magnitude-based pruning, not against the model you had. Nearly twice as good as a bad baseline, seventeen points below where you started.
The load-balancing loss does homogenize, the homogenization has now been measured under control, and it lands on router geometry rather than on the experts themselves. Its size is set by a coefficient chosen during a training run you did not do, and the spread between Ξ»_aux = 0.9 and Ξ»_aux = 0.001 is wide enough to decide whether the router probabilities in front of you carry usable information at all.
Underneath that router the experts are specialized enough that half of them do not carry the routing mass, and specialized in a way that does not transfer across domains, so a single general-purpose pruned model is a harder object than a domain-pruned one. Deleting experts buys memory and, on the one benchmark that measured all of this on the same hardware, essentially no throughput, while quantization bought more memory, better throughput, and cost nothing measurable in quality. Perplexity is an unsafe selection signal for any of it, and in the over-dispersed regime it is not merely noisy but inverted: the kind of proxy that keeps moving after it has stopped tracking the thing it stood for.
Whether a frontier model trained without an auxiliary loss prunes differently from an auxiliary-loss-trained peer of the same size is, as far as I could find in the 2026 literature, untested. The controlled evidence stops at 1B. That is the experiment I would want, and it is expensive in exactly the way that means nobody has run it yet.
βHow many experts can I deleteβ is two questions asked as one. Which axis am I buying, memory or compute. And how far do I trust the router that is telling me which experts matter. Both September papers answer the second the same way: less than you were about to.
References