Speculative decoding goes tree-based: DARTree reports up to 9.73x speedups

One-Line Summary

New inference-time methods speed and steer generation: a tree-based speculative decoder, a sliding recurrent memory, and a thought-level beam search.

Research Papers

DARTree turns speculative decoding into a parallel tree

To make large language models (LLMs) generate faster without changing their answers, DARTree proposes many next-token candidates at once and verifies them along a tree instead of a single chain. This extends speculative decoding by turning the draft step into a fixed-width tree that can be checked in parallel while keeping outputs identical to standard decoding. ¹

DARTree is training-free: it reuses a pretrained autoregressive (AR) correction head and generalizes it from sequential chains to trees. The method expands and scores all nodes at each depth in a single batch, then applies best-first pruning to select the verification tree, decoupling AR-head inference from sequential heap operations. ¹

Across seven math, code, and chat benchmarks and four model–temperature settings, DARTree reports the highest average acceptance length and speedup. It accepts up to 12.97 tokens per verification round — 98.6% more than DFlash and 27.9% more than Domino in the same setup — and reaches up to 9.73x lossless speedup over locally measured autoregressive decoding. ¹

Because candidate expansion is batched at fixed width and the correction head is reused, the design fits GPU-friendly batching and avoids extra training, positioning it as a practical upgrade wherever speculative decoding is already in place. ¹

Maglev pairs a teacher and decoder for sliding memory

Maglev introduces a Transformer architecture with fixed-size memory by training a more expressive “prefiller” to teach a sliding-window decoder what to remember for the next token. The prefiller Q uses interleaved full and sliding-window attention to produce memory targets, while the decoder P uses only sliding-window attention with recurrent key/value (KV) injection for next-token prediction. ²

A memory consistency loss aligns the decoder’s memory with the teacher’s, enabling inference with P alone. Empirically, Maglev improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent baselines, and parameter sharing between P and Q reduces parameter memory while preserving most of the gains. ²

Gambit focuses compute with thought-level beam search

Gambit treats multi-step reasoning as a search over partial thoughts and concentrates compute on the most promising ones, pruning weaker paths and immediately branching from strong prefixes. Instead of running many full traces independently, it uses a lightweight scorer on hidden states to keep hardware utilization high under a fixed budget. ³

Under identical hardware constraints, Gambit reports up to a 6.7% absolute accuracy gain on HMMT-24 and 3.3% on AIME-25 over pruning baselines, more than 2× higher throughput on trace completion, and up to 68.5% fewer tokens than standard parallel sampling. ³

Why It Matters

These papers converge on inference-time efficiency: tree-based propose-and-verify for faster generation (DARTree), compact fixed-size memory for long context (Maglev), and compute steering for multi-step reasoning (Gambit). The shared idea is to schedule the same hardware more intelligently to gain speed, accuracy, or memory headroom without retraining the base model. ¹²³

Operationally, the key variables to watch are acceptance length per verification round (DARTree), decoder–teacher memory alignment (Maglev), and scorer overhead versus throughput (Gambit). Real-world value will hinge on how these trade-offs behave under batched serving, kernel overheads, and long-context workloads. ¹²³