Machine learningvLLM BlogAug 23 2mo2 months agoExploring Speculative Decoding in vLLM on AMD GPUsA practical guide to speculative decoding in vLLM on AMD GPUs, covering draft-and-verify mechanics, MTP, EAGLE-3, DFlash, DSpark, configuration, tuning, and benchmark results.vLLMRead
Reliability and performancevLLM BlogAug 22 2mo2 months agoLarge-Scale Sharded Weight Transfer with Ray Direct Transport (RDT) in vLLMWe implement a native sharded weight transfer engine in vLLM utilizing Ray Direct Transport (RDT), achieving weight transfer for the Kimi K2 model in BF16 on 48 8xH100 nodes in 7.53svLLMRead
AI and agentsvLLM BlogAug 21 2mo2 months agoIsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRLIsoExec unifies numerical execution across SkyRL's vLLM and Megatron runtimes, reducing the average rollout-versus-training logprob difference below 1e-6 on Qwen3.5-35B-A3B with 25% overhead.vLLMRead
Reliability and performancevLLM BlogAug 20 2mo2 months agoVeRL-Omni v0.2.0: Faster Diffusion RL and Stable Omni TrainingA release focused on higher-throughput diffusion rollout, reusable omni adapters, and broader recipe coverage.vLLMRead
Reliability and performancevLLM BlogAug 17 2mo2 months agoDistributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-OmniDistributed Layerwise Offload shards and streams DiT weights across devices, serving a measured 124 GB Cosmos3 model on 64 GB HBM and estimating a path toward 200B+ models.vLLMRead
Reliability and performancevLLM BlogAug 14 2mo2 months agoAdaptive Verification in vLLM: DSpark confidence-scheduled verificationSizing the DSpark draft-verification budget from per-request confidence instead of verifying every drafted token, so one configuration holds the throughput/latency frontier from batch size 1 to 256.vLLMRead
InfrastructurevLLM BlogAug 12 2mo2 months agoDay 0 Support for Qwen3.8-2.4T-A95B on vLLMDay-0 vLLM support for Qwen3.8-2.4T-A95B: a 2.4-trillion-parameter hybrid MoE model served out of the box, with FP8/BF16 checkpoints plus NVFP4 and MXFP4 quantized weights, and co-developed kernels on NVIDIA and AMD hardware.vLLMRead
Blogsby Ibis