AI and agentsvLLM BlogJun 29 3mo3 months agoMicro-Agent: Beat Frontier Models with Collaboration inside Model APIHow vLLM Semantic Router turns vllm-sr/auto into a bounded micro-agent runtime for Confidence, Ratings, ReMoM, Fusion, Workflows, and benchmark-shaped collaboration.vLLMRead
AI and agentsvLLM BlogJun 23 4mo4 months agoEngineering TTS Inference in vLLM-OmniHow vLLM-Omni supports and optimizes Qwen3-TTS, VoxCPM2, Higgs Audio V3, and Fish Speech S2 Pro with staged serving, batching, CUDA Graphs, and model-specific kernels.vLLMRead
Product and companyvLLM BlogJun 16 4mo4 months agoBeyond One Model: Fusion in vLLM Semantic RouterHow vLLM Semantic Router Fusion runs a panel of models, uses a judge to analyze agreement and gaps, and synthesizes one answer while preserving routing policy, traces, and OpenAI-compatible serving.vLLMRead
Product and companyvLLM BlogJun 12 4mo4 months agoMiniMax M3 in vLLM: Day-0 Serving for 1M-Token Multimodal ReasoningHow vLLM serves MiniMax M3 with MiniMax Sparse Attention, multimodal and reasoning parsers, MXFP8 weights, and long-context deployment recipes.vLLMRead
AI and agentsvLLM BlogJun 10 4mo4 months agoDiffusionGemma: The First Diffusion LLM (dLLM) Natively Supported in vLLMHow vLLM supports DiffusionGemma, the first native diffusion language model in vLLM, using Model Runner V2 state hooks, iterative denoising, bidirectional attention, and reused speculative decoding paths.vLLMRead
Machine learningvLLM BlogJun 9 4mo4 months agoAnnouncing vime: A Simple, Stable, and Efficient RL Framework for LLMsvime connects slime's training stack with vLLM rollouts to provide a simple, stable, and efficient RL post-training pipeline.vLLMRead
InfrastructurevLLM BlogJun 5 4mo4 months agovLLM Semantic Router v0.3 Themis: From Signals to Stateful Production RoutingWhat vLLM Semantic Router v0.3 Themis adds for production routing: canonical config, inspectable signal-decision-policy flows, safer operations, CLI/dashboard/Kubernetes alignment, and replayable routing behavior.vLLMRead
Product and companyvLLM BlogJun 4 4mo4 months agoAnnouncing Day-0 Support for NVIDIA Nemotron 3 Ultra on vLLMHow to serve NVIDIA Nemotron 3 Ultra with vLLM for long-running agentic reasoning, including BF16 and NVFP4 checkpoints, supported GPU configurations, OpenAI-compatible deployment, and NeMo RL integration.vLLMRead
AI and agentsvLLM BlogJun 3 4mo4 months agoFast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AIWhat the DeepLearning.AI vLLM course teaches: optimizing, deploying, and benchmarking LLM inference with LLM Compressor quantization, GuideLLM, KV cache sizing, serving, and memory tradeoffs.vLLMRead
AI and agentsvLLM BlogJun 2 4mo4 months agoAccelerating vLLM-Omni Inference with AutoRound QuantizationHow AutoRound integrates with vLLM-Omni to serve W4A16 quantized multimodal, diffusion, image, and video models with smaller checkpoints, preserved quality, Intel XPU acceleration, and NVIDIA GPU support.vLLMRead
AI and agentsvLLM BlogJun 2 4mo4 months agoSession-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM AgentsHow Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cache-aware switch pricing, and replayable traces.vLLMRead
Blogsby Ibis