What Happened

Inference is the bottleneck . Agents read, plan, , . They consume tokens at a approached. tokens takes a full forward . , we are building the inference stack scaled economics of tomorrow. a sneak peek.

Our team released DFlash in January; SGLang, vLLM, TensorRT-LLM, and llama.cpp. NVIDIA measured up to 15× throughput Blackwell GPUs; Google reported 3× more tokens per second on TPUs; CoreWeave's production Kimi K2.7 Code endpoint, the fastest Artificial Analysis, runs DFlash by default. The ecosystem now builds on it: NVIDIA, Red Hat, published DFlash drafters; Meta (Muse Glimmer), Poolside (Laguna), Xiaomi (MiMo-V2.5-Pro), and NVIDIA (Nemotron 3.5 Lightning) ship official drafters models. On Hugging Face, DFlash models have been downloaded more than 3.5 million times (as of August 2026).

Speculative decoding is a modern inference stack.1 A guesses a block of tokens, and the target model verifies forward pass. Good guesses several tokens; thrown away. For years, though, the draft itself stayed autoregressive: a time. DFlash -pass too: the entire block, every position, predicted in parallel.

DFlash 2 pushes parallel drafting one step further: over 20% more output from every verification pass, for around 1% added cycle latency, with the output provably unchanged. Across benchmarks 16–25%. With the Qwen3.8-27B drafter released today, SGLang serves at 2.7–3.4× the throughput of autoregressive decoding 1. Predicting every position independently leaves headroom in two places: choosing the right tokens and holding accuracy . DFlash 2 recovers both without giving -pass design.

Why It Matters

pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python" python -m sglang.launch_server \ --model-path Qwen/Qwen3.8-27B \ --speculative-algorithm DFLASH \ --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \ --speculative-num-draft-tokens 8pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head" /Qwen3.8-27B \ --speculative-config '{ "method": "dflash", "model": "incoai/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7 }'://github.com/ggml-org/llama.cpp.. origin pull/27342/head:pr-27342 git switch pr-27342 # NVIDIA CUDA cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build -j # Apple Silicon cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON cmake --build build -j ./build/bin/llama-server \ -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \ -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \ --spec-type draft-dflash \ --spec-draft-n-max 7Download and install the prebuilt oMLX with DFlash 2 support.

Manager -community/Qwen3.8-27B-4bit. Configure DFlash with the following settings:

DFlash predicts every position independently, in parallel. plausible . Yet nothing together, and an incoherent verification. Recent methods such as Domino and DSpark buy coherence with sequential heads that rewrite each position's full-vocabulary distribution. costly autoregressive correction really necessary?

No. The evidence is already in DFlash's own candidate lists. position: DFlash's 85.4% , 16 candidates 99.5% . , usually .

What Comes Next

An oracle that always candidate 16 acceptance length from 4.27 to 6.79. selection headroom. select through the candidates.

Diffusionisgood⟨mask⟩⟨mask⟩⟨mask⟩Independent Top-1 Picksfor✗same word, twiceposition 1decodingspeculativeslow⋮position 2decodingthinkingmodels⋮position 3⟨eos⟩again…⋮all adjacent pairs scored at once → one path keptaccepted outputfortarget-decoded tokenmask tokenaccepted draftselected path A Lightweight Path Selector Coherence is mostly local: a candidate's fit depends mainly before it, so scoring neighboring pairs should be enough. DFlash 2 16 candidates at each position and scores every adjacent pair: for predecessor aaa and current candidate bbb,

. The first, Ut(b)U_t(b)Ut​(b), is DFlash's own logit: drafter already . The second follows aaa: a compact 256-dimensional embedding, embeddings are matched under a context gate H(ht)H(h_t)H(ht​) that decides . In essence, this is a low-rank bilinear attention over adjacent candidates.

Scoring stays fully parallel. Every adjacent position is scored , backbone or LM-head pass. The only sequential precomputed scores: starting verified token, greedy follows the best successor , sampling scores, and rejection sampling restores the exact target distribution.

The selector improves DFlash by 0.34 tokens at T=0T=0T=0 and 0.47 at T=1T=1T=1. DSpark correction in both settings with roughly 40× fewer parameters and 16× lower latency overhead. Choosing is cheaper than predicting. : the oracle reaches 6.79. Pairwise scoring is the simplest selector , and we believe there is plenty to explore.

We also noticed both recall rows above decline toward . Even the oracle decays: with perfect selection, accuracy 99.5% position to 87.8% . No selector , because the candidates themselves are running out. suffix decay, a backbone problem.

One suspect is capacity: a five-layer backbone preserve dependencies across the block. , depth should positions. ! 3-, 5-, and 15-layer DFlash models are almost identical position, . indiscriminate: ten extra attention blocks add capacity everywhere, positions that had little , efficiency that makes DFlash attractive.

Explore more: Software & AI Guide