Mixture-of-experts models activate few parameters per token but retain every expert in memory, making deployment capacity very different from compute capacity. This study tests a compression sequence rather than an isolated technique: rank experts using router-weighted activation evidence, prune the low-saliency half, heal the surviving function, and only then quantize. On Qwen3-30B-A3B, the 50%-pruned model moved from 0.518 ARC before healing to 0.524 after full-function healing, versus a 0.522 baseline. Simulated NVFP4 then produced 0.510 ARC and 2.63 perplexity versus the baseline's 2.99. The composition is promising but not yet a deployment claim: the experiment used masked rather than structurally removed experts, simulated quantization, and a limited evaluation suite. The active extension moves to physically pruned REAP50 artifacts and asks whether routing and activation statistics can assign multiple precision tiers among the surviving experts.
1. Research question
Expert pruning and weight quantization solve different parts of the MoE deployment problem. Pruning removes entire functions and reduces the number of expert slots. Quantization retains the functions but reduces the cost of each slot. Applied independently, either technique can consume the quality budget before the model reaches a useful memory footprint.
REAP changes the pruning signal from frequency alone to router-weighted expert activation: an expert matters when the router sends it meaningful weight and its output has meaningful magnitude. The first question is whether the surviving network can be healed enough that low-bit quantization becomes a small, additive loss rather than a compounding one.
Does the order prune → heal → quantize preserve more generative function than treating expert count and precision as independent compression knobs?
The next question is finer-grained: after pruning, should every surviving expert receive the same precision? A uniform format ignores the fact that experts differ in routing frequency, output impact, domain specialization, and spill cost.
Compression as a sequence of decisions
2. Completed protocol
Subject
Qwen3-30B-A3B, a 128-expert sparse MoE, provides a tractable intermediate scale: large enough for substantial expert redundancy, but small enough to run controlled pruning and healing experiments.
Pruning signal
For each routed token, the experiment combines the router gate value with the norm of the expert output. Scores are aggregated per expert and the lowest-saliency half are removed from routing. The shared expert is not pruned.
Healing
The decisive arm updates the surviving model function, not only the router. A 600-step paged 8-bit AdamW run heals experts, attention, norms, router, embeddings, and output head. Masked experts receive no useful gradient. This is still a functional experiment; physical tensor slicing is a separate artifact step.
Quantization
The healed survivors are passed through an NVFP4 simulation with block-scaled 4-bit weights. The study then re-runs ARC, ARC-Easy, and perplexity using the same evaluation harness and split.
3. Preliminary results
| Arm | Experts | ARC | ARC-Easy | Perplexity ↓ |
|---|---|---|---|---|
| Baseline | 128 | 0.522 | 0.792 | 2.99 |
| REAP prune 50% | 64 | 0.518 | 0.787 | 3.12 |
| + full-function heal | 64 | 0.524 | 0.804 | 2.61 |
| + simulated NVFP4 | 64 | 0.510 | 0.785 | 2.63 |
Each arm against the unpruned baseline
At this scale, one-shot 50% REAP pruning is already close to baseline. Full-function healing restores the small loss and slightly exceeds the baseline on all three reported metrics. Adding NVFP4 costs 0.014 ARC relative to the healed arm and finishes 0.012 below baseline, while perplexity remains better than baseline.
The useful inference is narrow: in this setup, post-heal quantization adds a small loss rather than reopening the full pruning loss. It does not prove that the pipeline is near-lossless on a different MoE, at a deeper prune ratio, or under a generative benchmark suite.
4. Failed arms that shaped the protocol
Router-only healing moved the 50%-pruned ARC score from 0.518 to 0.510. Routing could not reconstruct expert function that was no longer present.
The healed 75% arm reached 0.464 ARC versus 0.522 baseline. The degradation was graceful, not catastrophic, but too large for the intended artifact.
OLMoE-7B lost heavily at 50%, while Qwen3-30B-A3B remained near baseline. Expert redundancy did not transfer cleanly across scale.
5. Active extension: multiple precision tiers
The completed arm quantizes every surviving expert uniformly. The next study asks whether a fixed memory budget is better spent non-uniformly. Candidate signals include:
- router frequency and accumulated gate mass;
- router-weighted output magnitude, aligned with the pruning signal;
- domain-conditional specialization and calibration-set drift;
- sensitivity of target logits to expert-local quantization noise;
- residency and spill cost in the actual inference runtime.
The first deployment target is a physically pruned Hy3 REAP50 artifact. The checkpoint and CPU-only loader path are prepared: the artifact retains 96 routed experts across MoE layers and uses top-8 sigmoid routing. Runtime quality and performance remain explicitly unverified pending target-rig gates. That distinction matters—the existence of a loader is not evidence that the compression works.
6. What this does not establish
- The Qwen experiment masks experts; it does not yet measure the memory or latency of a physically compacted checkpoint.
- NVFP4 is simulated in the completed study, so kernel and layout effects are outside the result.
- ARC, ARC-Easy, and perplexity are not sufficient for code, tool-use, or long-form generative claims.
- The multi-precision policy is a hypothesis and figure annotation, not a completed result.
- Hy3 loader preparation is CPU-only; no GPU quality or throughput conclusion is claimed.
7. Artifacts and release plan
The public runtime work lives in bw24, including the model-loading and expert-spill machinery used for the next deployment phase. Experiment scripts, masks, healed checkpoints, and the evaluation ledger are being prepared for a standalone release with the first complete paper version.
- Available now Method, completed Qwen table, failed arms, limitations, and extension protocol.
- Next release Physical expert compaction, checkpoint manifest, generative evaluation, and runtime measurements.
- Paper v1.0 Hy3 replication, multi-tier precision ablation, multi-corpus calibration, PDF, and BibTeX.
References
- M. Lasby et al. REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression. 2025.
- S. Jha et al. REAM: Merging Improves Pruning of Experts in LLMs. 2026.
The Hy3 and mixed-precision sections are marked as open until a target-rig experiment exists. Loader completion, quality, and throughput are three different milestones.