vLLM: Easy, Fast, and Cheap LLM Serving for Everyone

vLLM: Easy, Fast, and Cheap LLM Serving for Everyone - Simon Mo, vLLM vLLM is an open source library for fast, easy-to-use LLM inference and serving. It optimizes hundreds of language models across diverse data-center hardware—NVIDIA and AMD GPUs, Google TPUs, AWS Trainium, Intel CPUs—using innovations such as PagedAttention, chunked prefill, multi-LoRA and automatic prefix caching. It is designed to serve large scale production traffic with OpenAI compatible server and offline batch inference, scalable to multi-node inference. As a community-driven project, vLLM collaborates with foundation model labs, hardware vendors and AI infrastructure companies to develop cutting-edge features. In this talk, I will introduce the vLLM project, it’s technical aspect, recent update, and upcoming roadmap.