serving-llms-vllm

The serving-llms-vllm skill enables high-performance deployment of large language models (LLMs) using vLLM, a serving engine designed for production-scale inference. It addresses common challenges in LLM deployment such as limited GPU memory, low throughput, and high latency. By leveraging vLLM’s PagedAttention mechanism, which uses a block-based KV cache, and continuous batching, which dynamically mixes prefill and decode requests, this skill significantly increases throughput compared to standard transformer-based serving approaches. It is intended for teams deploying LLM-powered APIs that require reliability, scalability, and efficient hardware utilization.

Key capabilities include support for OpenAI-compatible HTTP endpoints, enabling seamless integration with existing OpenAI SDK-based applications. The skill supports multiple quantization methods such as GPTQ, AWQ, and FP8 to reduce memory usage and improve performance, as well as tensor parallelism for serving large models (e.g., 30B–70B parameters) across multiple GPUs. It also provides production-oriented features such as configurable GPU memory utilization, prefix caching, Prometheus-compatible metrics exposure, and Docker-based deployment. Offline batch inference is supported through the Python API, allowing efficient processing of large prompt datasets without server overhead.

Typical use cases include deploying chat or completion APIs, optimizing inference latency and throughput for customer-facing applications, and running large-scale batch processing jobs. This skill is suitable for machine learning engineers, MLOps practitioners, and backend developers responsible for operating LLM services in production environments. It is particularly valuable when serving large models under GPU memory constraints or when high request concurrency and observability are required.