Research

Negative-eigenvalue gates for hybrid models

Qwen3.5 and Qwen3.8 are hybrids: the 27B has 48 Gated DeltaNet layers and 16 attention layers. A Gated DeltaNet update scales the state along the key direction by 1 − β, and with β limited to (0, 1) that factor can shrink a direction but never flip it. State tracking such as parity needs the flip. Widening β to (0, 2) allows negative eigenvalues. The retrofit adds that range to a pretrained model through a zero-initialized gate, so at step zero the model is bit-identical to the original, and trains it with a LoRA.

4B, tier-1 state tracking at 4x length, over LoRA alone (3 seeds)

+48.3 pt

9B, same tasks (3 seeds)

+43 pt

Qwen3.8-27B, mostly parity (1 seed)

+16.1 pt

on-call agent probe, LoRA arm, thinking off

65.7→83.1%

At 4B the confidence interval is 38.4 to 54.1 points on the tier-1 tasks (parity, swap, codeswap). On the 27B the first stage showed the mechanism but no workload gain: the two arms scored 93.2 and 93.6 on the session benchmark, both up from 69.8 for the untouched model, so the second stage did not run.

The LoRA carried forward. On the on-call agent probe it lifted accuracy from 65.7% to 83.1% with thinking off, at 3.9k tokens a task. With thinking on it matched the untouched model with 27.7k fewer reasoning tokens a task. The current run trains both arms on thinking-on data at 27B and measures accuracy and reasoning tokens per task.

The 27B tier-1 number is one seed. Whether the gate pays on real agent work at 27B is the open question this run answers.

Layby: where an idle session's KV cache should wait

When an agent or chat session goes idle, the engine has to decide where its KV cache waits: GPU memory, CPU memory, disk, or nowhere, to be recomputed later. Layby makes that call per session. Layby-Dwell, a 421M-parameter encoder fine-tuned from Laya, reads what the server saw at the end of a response and returns a calibrated curve of the time to the next request. A cost rule turns the curve into one of four placements. Every term is a probability from the curve times a cost the engine measures on itself while it runs. There are no fitted constants.

p95 time to first token on resumed turns, against an LRU CPU tier, Qwen3-8B, slow disk

0.59x

p95 against the better of two LMCache settings, same runs

0.49x

p95 against an LRU CPU tier, GLM 5.3 Flash, heavy load, p99 held at the baseline

0.79x

Measured on vLLM 0.31 on Nebius RTX PRO 6000 nodes with a network disk read without the page cache, three repeats for Qwen3-8B (median latency unchanged at 1.01x) and two 4-GPU engines on one 8-GPU node for GLM 5.3 Flash at heavy load (at half load it ties the LRU tier). On that disk, writing every session to disk is worse than no disk at all: 1.61x and 1.35x at p95. A simulator registered its prediction for a shared disk before the disk arms ran; the first node's GLM ratios landed within 0.13 of it. On ReturnBench, an open benchmark of 33 settings built from public traces, Layby is at 0.66x of the LRU tier (geometric mean) and never worse than 1.04x in any setting.

What did not work: on A100 boxes whose disk tier was really the page cache, writing everything was as good or better with the first adapter (0.57x against Layby's 0.69x on the private agent replay; 0.15x against 0.16x on the public one). With the fixed adapter, on two of three boxes Layby tied writing everything (0.89x of it at p95, 0.64x of the LRU tier); on a slower box with a 0.54 GB/s disk it lost (13.5 s against 4.5 s at p95). The cause is not identified yet. The rule chose about the same mix as on the fast boxes (no disk copy for 613 to 616 of 653 turns on every box), and CPU memory still held 0.99 of idle KV at 300 s, so eviction does not explain the loss. It will be rerun with full telemetry on a host of the same class. A pre-emptive return-time guard, write-back on eviction and a forward-looking wait cap all failed their bar and are reported as negative results. Several engine-integration bugs made a correct rule lose until fixed; each fix ships with a test that fails before it.

Paper on arXiv to follow. The fixed-adapter A100 rerun is one run per arm on each of three boxes. A full re-evaluation on stable hosts has started, beginning with checks that need no GPU.

Layer-loop weight sharing needs residual scaling, and then beats vanilla when data is the limit.

I combined two results that had not met: layer-loop weight sharing, shown with unscaled residuals, and looped-residual scaling theory, analyzed on model-level reuse. The correlated-accumulation problem is real and does not depend on the mixer. I showed it numerically and then in from-scratch pretraining: the gain from the ε=λ/(N√L) fix follows 0.0122·(N−1) with R²=0.993, and it pins the best learning rate at its N=1 value. Under fixed unique tokens (about 20 epochs) at d=768, the looped and scaled model beat the FLOPs-matched vanilla model on best validation loss on 3 of 3 seeds (−0.039±0.003) with 34% fewer parameters. It held in all three mixer families, seven paired runs with no sign flips, and the hybrid rounds isolated a rule: loop the state mixer, never the retriever.

An MQAR probe bounded the mechanism: loops multiply what recurrent state holds. They do not replace attention recall.

The memory-layer retrofit is free. Its forgetting advantage never reached significance.

I grafted a product-key memory layer onto a stock dense Qwen3-4B and ran five sequential update rounds against LoRA, a regime the published 0.5B single-shot retrofits never tested. The capability cost of the retrofit turned out to be the routing warmup's step size. At lr/10 the graft costs nothing on arithmetic and format, comes out slightly ahead on code and trivia, and still reaches full acquisition by round three. The forgetting advantage failed to appear in 12 of 12 probe-arm confidence intervals, including a reference-faithful reproduction of Meta's parameterization.

One model, one graft layer, synthetic facts. The five-round regime showed nearly the full effect by round one.

Imatrix-aware scale search recovers NVFP4 quality. Hadamard rotation does not.

I tested two levers for NVFP4 weight quantization. Brute-force scale search with importance-matrix weighting cut perplexity by 6.0% on Qwen3.5-9B, and held on a 26B MoE and on two corpora. Hadamard rotation, the standard recovery trick for other formats, made every tensor worse (mean relative MSE 3.4 to 4.1% higher), checked against a bit-exact C oracle over 50 million real weights.

The ranking flipped with model size: at 0.5B scale search dominates, at 9B the imatrix does. Sub-1B quantization results did not carry over.

Block-routed SwiGLU: the kernel is nearly free, the capability is not there.

I prototyped input-dependent re-pairing of gate and up channels inside a standard SwiGLU MLP, adding routing without touching the two large matrix multiplications. The Triton kernel passed its cost check on H100: 0.73 to 0.83% more block latency, safe under CUDA graphs, no extra peak memory. Seeing that took interleaved measurement, because the naive benchmark inflated the overhead 5 to 10x. The equal-parameter capability test then refuted the idea: 63.4% against the baseline's 66.1% on held-out factor compositions across five seeds.

NVFP4 compression alone did not degrade the co-trained MTP head.

The hypothesis predicted hidden-state shift: quantize the trunk, keep the co-trained MTP head more precise, and draft agreement should fall, especially deeper into the context. After removing the confounds, it did not.

The first result looked like damage and was a measurement trap. The lab note walks through it.