I work at the boundary between model architecture and inference systems: speculative decoding, draft-head compression, expert pruning, quantization, and deployment under fixed memory and power budgets.
Publication policy
“Working paper” means the question and protocol are stable enough to inspect. It does not imply peer review, a venue, or a finished result.
Working papers / 01
The working papers.
The papers grow from live experiment ledgers. Each carries its version and status; a concluded paper keeps its failed arms visible.
A compact MTP block with its own hot-token vocabulary, trained against the real deployment regime. The audit answered: function cuts pay a 10–19 point off-distribution tax that fidelity cuts do not, and the zero-training trimmed-vocabulary recipe survives.
Router-weighted expert pruning preserves the router's freedom but removes function. The study asks how much full-function healing restores, how quantization composes with that recovery, and where mixed precision should go next.
The memory-layer retrofit is free; its forgetting advantage never reached significance.
Grafted a product-key memory layer post-hoc onto a stock dense Qwen3-4B and ran five sequential update rounds against LoRA—the regime the published 0.5B single-shot retrofits never touched. The supposed capability cost of the retrofit turned out to be the routing warmup's step size: at lr/10 the graft costs 0% on arithmetic and format, is negative on code and trivia, and still reaches full acquisition by round three. The headline forgetting advantage failed to appear in 12 of 12 probe-arm confidence intervals, including a reference-faithful reproduction of Meta's parameterization.
Capability cost of the routing warmup, by probe (mean of 3 seeds). The entire "retrofit tax" was the warmup's step size: at lr/10 every probe sits at or below zero cost while acquisition still reaches 1.000 by round three.
Evidence
Three configurations × three seeds, four retention probes, per-arm learning-rate calibration, acquisition gating. Four earlier verdicts were self-overturned—context copying, a dense update in sparse clothing, an unseeded RNG—and the superseded runs stay in the repo as receipts.
Decision
Power analysis says n=3 could never have resolved the claim (required n runs 2–621 per probe). A ~30-seed-per-arm run, and the 3× code-variance drop under the reference config, are what can change the status.
One model, one graft layer, synthetic facts; the five-round regime showed nearly the full effect by round one.
Layer-loop weight sharing needs the ε=λ/(N√L) fix—and then beats vanilla in the data-constrained regime.
Combined two results that had never met: layer-loop weight sharing (proved with unscaled residuals) and looped-residual scaling theory (analyzed on model-loop reuse). The correlated-accumulation pathology is real and mixer-agnostic—proved numerically, then confirmed in from-zero pretraining: the fix's gain follows 0.0122·(N−1) with R²=0.993 and pins the optimal learning rate at its N=1 value. The centerpiece: under fixed unique tokens (~20 epochs) at d=768, the looped+scaled model beat the FLOPs-matched vanilla on best-achievable validation loss on 3 of 3 seeds (−0.039±0.003) with 34% fewer parameters. The flip held in all three mixer families—seven paired runs, zero sign flips—and the hybrid rounds isolated the rule: loop the state-mixer, never the retriever.
Best-achievable validation loss at d=768, data-constrained (~20 epochs over fixed unique tokens), three seeds per arm. The looped model wins every seed pairing with 34% fewer parameters; the seed spreads never overlap.
Evidence
Eleven pre-registered GPU rounds plus five CPU numerical proofs; a no-single-seed rule that forced one retraction of the campaign's own headline; the fresh-data crossover claim refuted at every N and both mixer families.
Decision
The recipe holds where data, not compute, is the wall. The d≥1024 rung and the remaining hybrid seed legs decide whether the flip keeps growing with scale.
An MQAR probe bounded the mechanism: loops multiply what recurrent state holds; they do not replace attention recall.
Block-routed SwiGLU: the kernel is nearly free; the capability isn't there.
Prototyped input-dependent gate/up channel re-pairing inside a standard SwiGLU MLP—routing added without touching the two dominant matrix multiplications. The Triton kernel passed its cost gate on H100: +0.73–0.83% whole-block latency, CUDA-graph safe, zero extra peak allocation—after interleaved measurement exposed a 5–10× overhead inflation in the naive benchmark. The equal-parameter capability test then refuted the motivating hypothesis: 63.4% vs the baseline's 66.1% on held-out factor compositions across five seeds.
Evidence
Correctness matched the PyTorch reference; both arms sat at chance on the random-rule control, so the test measured composition, not memorization.
Decision
Cheap routing machinery is real; this routing rule earned no language-model run. The kernel and the negative are both public.
Negative result · closed 10 Jul 2026
NVFP4 compression alone did not degrade the co-trained MTP head.
The original hypothesis predicted hidden-state shift: quantize the trunk while keeping the co-trained MTP head relatively precise, and draft agreement should fall—especially with context depth. After removing confounds, it did not.
Evidence
Forced replay to 64k, generation-distribution replay, and seeded sampled live runs across two clean safetensors quantization recipes.
Decision
Close the compression-only healing premise. Keep live-chain drift separate. Redirect the project toward compact heads and harsher quantization regimes.
Documented in the experiment ledger with per-arm artifacts; the evidence matrix ships with the paper's first complete release.
Positive result · in review at llama.cpp
Imatrix-aware scale search recovers NVFP4 quality; Hadamard rotation does not.
Two levers were tested for NVFP4 weight quantization. Brute-force scale search plus importance-matrix weighting cut perplexity by 6.0% on Qwen3.5-9B and held across a 26B MoE and two corpora. Hadamard rotation—the standard recovery trick at other formats—made every tensor worse (−3.4% to −4.1% mean relative MSE), verified against a bit-exact C oracle over 50 million real weights.
Evidence
Three models, two corpora, end-to-end wikitext-2 perplexity; rotation checked with orthonormality invariants and a compiled ggml oracle at zero mismatches.
Decision
Ship scale search and imatrix support; close the rotation premise for this block format. Both are implemented in the llama.cpp pull request, along with wiring for the previously dead NVFP4 file type.
A scale reversal appeared with model size: at 0.5B the scale search dominates, at 9B the imatrix does. Sub-1B quantization conclusions did not generalize upward.
Open programs / 03
Questions with apparatus built and receipts pending.
Each of these has a written protocol and public code; the deciding runs are what remains.