Your Experts Have Partners. There's HOPE for Keeping Them Together
Your Experts Have Partners. There's HOPE for Keeping Them Together
Higher-Order Pruning of Experts in Mixture-of-Experts Language Models
Introduction
Mixture-of-Experts (MoE) models have enabled state-of-the-art LLMs to be scaled to hundreds of billions of parameters, but they come with a fundamental and costly caveat: every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and 95% of its ~250 GB parameter memory footprint is taken up entirely by these expert MLPs alone. However, for any given token, only 8 experts in each layer are actually used.
Expert pruning is one of the most direct fixes to this memory bottleneck. That is, permanently delete the least useful experts. Pruning avoids the latency issues which make dynamic loading/off-loading of experts impractical; it does not require any training; and it composes with quantization. Given these attractive benefits and the simplicity of the idea, it's easy to see why expert pruning has become a go-to approach for shrinking MoEs. The difficulty lies in deciding which experts are safest to remove.
Existing pruning methods score each expert independently, assigning every expert a scalar importance score (based on a calibration dataset), and cutting out the least important experts (in each layer independently). These are first-order methods, because they assume each expert's contribution is independent of all others. In reality, experts form cooperative groups. We observe consistent clusters of co-activated experts in our own experiments, and independent work has shown that expert cooperation is a stable property of MoEs and is important for model performance. Ignoring these interactions leads to suboptimal pruning, especially at aggressive rates.
Our newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. At 50% pruning, HOPE achieves the best average rank of any method, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method.
In this post, we explain how HOPE works, show where it outperforms existing methods, and walk through how to use it for your own models. For readers unfamiliar with MoE architectures, see Mixture of Experts (MoEs) in Transformers.
Experts don't work alone
In a standard MoE, each layer has experts, and each input token is routed independently to such experts ( is a typical choice). Routing is performed by a single layer which produces softmax-normalized gate weights for each expert , wherein the experts with the highest gate weights are selected to form the active set of experts, . The output of the layer is then , where is the output of expert for token .
Left: typical MoE layer, where each token is routed to the top-K experts. Right: first-order methods rank and delete experts based on a scalar importance score, but the best pruning decisions will also consider the strength of interactions between experts.
Importantly, experts in the selected set are not independent. Experts form stable cooperative groups that get consistently co-activated together. This has been observed before in the literature, as well: expert co-selection patterns emerge early in training and stabilize throughout.
This means that when we prune experts from an MoE layer, pruning decisions need to consider the higher-order effects between these experts. Removing expert A might seem fine on its own, but if expert B—which frequently cooperates with and relies on A—is highly important, then removing A would dismantle a cooperative group just through a single deletion. First-order methods miss this entirely, because they score A and B separately.
This effect compounds at high pruning rates. When we remove up to 50% of experts, we risk breaking many cooperative links. This is where second-order pruning is most advantageous.
How HOPE identifies a prune-set
(See Appendix A in our corresponding paper for the full derivation of the HOPE objective)
When we delete a set of experts from some MoE layer, the selected set of experts (for any token ) can change. The pruned experts that would have been selected are replaced by the experts with the next-highest gate weights. The resulting change in layer output is the pruning error. Our goal is to find the prune-set (of some predefined pruning budget ) that minimizes the resulting pruning error (over tokens from our calibration set).
Unfortunately, minimizing the pruning error over directly is intractable. The pruning error depends on the identities of the replacement experts, which in turn depend on the prune-set itself. HOPE sidesteps this by defining a tractable surrogate , which strictly upper-bounds the dominant component of the pruning error:
Because is a squared sum, expanding it naturally produces cross-terms between pairs of experts. These are the interaction terms that first-order methods discard.
To find the optimal prune-set, HOPE minimizes over the calibration set, which can be reformulated as the following quadratic program:
