Avi Fenesh
Research archive
Updated 11 July 2026

Research, with the working state left visible.

I work at the boundary between model architecture and inference systems: speculative decoding, draft-head compression, expert pruning, quantization, and deployment under fixed memory and power budgets.

Publication policy

“Working paper” means the question and protocol are stable enough to inspect. It does not imply peer review, a venue, or a finished result.

Working papers / 01

Questions currently earning an answer.

The papers grow from live experiment ledgers. Numbers are labeled preliminary until the full protocol, controls, and replication pass are complete.

WP–01 / active
Speculative decoding · draft-head compression

Small-vocabulary MTP heads for memory-bound speculative decoding

A compact MTP block with its own hot-token vocabulary, trained against the real deployment regime. Current StudentSV experiments test whether reducing both the transformer block and output projection beats trim-only methods.

Status
Training curve active
Model
Qwen3.5-9B
Current signal
0.527 vs 0.675 code acceptance
↗
WP–02 / active
MoE · pruning · quantization

Prune, heal, quantize: composing expert pruning with low-bit MoE deployment

Router-weighted expert pruning preserves the router's freedom but removes function. The study asks how much full-function healing restores, how quantization composes with that recovery, and where mixed precision should go next.

Status
Protocol expanding
Model
Qwen3-30B-A3B → Hy3
Current signal
50% expert prune + heal + NVFP4 near baseline
↗
Result notes / 02
Negative result · closed 10 Jul 2026

NVFP4 compression alone did not degrade the co-trained MTP head.

The original hypothesis predicted hidden-state shift: quantize the trunk while keeping the co-trained MTP head relatively precise, and draft agreement should fall—especially with context depth. After removing confounds, it did not.

Evidence

Forced replay to 64k, generation-distribution replay, and seeded sampled live runs across two clean safetensors quantization recipes.

Decision

Close the compression-only healing premise. Keep live-chain drift separate. Redirect the project toward compact heads and harsher quantization regimes.

Documented in the experiment ledger with per-arm artifacts; the evidence matrix ships with the paper's first complete release.

Method / 03

What I expect a research page to disclose.

A result is easier to trust when the site makes it obvious what could still overturn it.

  1. 01

    Question

    A falsifiable claim, a target regime, and a reason the answer changes what gets built.

  2. 02

    Protocol

    Matched models, prompts, seeds, context lengths, baselines, noise floor, and success criteria written before the headline.

  3. 03

    Artifact

    Code, configs, model provenance, logs, and the exact path from checkpoint to reported number.

  4. 04

    Decision

    Supported, refuted, or still open—plus the failed arms and the next experiment that can change the status.

Collaboration · counterexamples

Have a result I should compare against?

Email Avi ↗ GitHub ↗