vLLM — Serve LLMs With an OpenAI-Compatible API

vLLM — Serve LLMs With an OpenAI-Compatible API

What it is. vLLM is an open-source inference and serving engine that turns a Hugging Face model into a fast, production-grade API server with one command — vllm serve <model-name> — using PagedAttention and continuous batching to handle many concurrent requests efficiently on GPU (and CPU/TPU backends). (github.com/vllm-project/vllm)

What it replaces. The hand-rolled Flask/FastAPI wrapper around a transformers.pipeline() call that chokes the moment two requests land at once — and, for teams currently paying per-token for a hosted LLM API just to get an OpenAI-shaped endpoint, the ability to run the same interface against your own GPU instead. vLLM’s server speaks the OpenAI Completions and Chat Completions API natively, so client code written for OpenAI’s SDK works against it unchanged.

What you can build. A local, OpenAI-compatible chat endpoint in two commands, no cloud key required:

pip install vllm
vllm serve Qwen/Qwen2.5-1.5B-Instruct

Then call it exactly like the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in one sentence."}]
  }'

Point any OpenAI-SDK-based app (LangChain, LlamaIndex, your own backend) at http://localhost:8000/v1 and it just works — swap the base URL, keep the code. (Commands verified against vLLM’s own quickstart docs.)

Where to get it. github.com/vllm-project/vllm — licensed under Apache License 2.0, confirmed directly from the repo’s LICENSE file, a permissive, genuinely open-source licence with no usage restriction. Docs at docs.vllm.ai. This is not a weekend toy: the project has over 2,000 contributors and is used as the serving layer inside production stacks at companies running LLMs at scale (per the project’s own README), and it supports NVIDIA, AMD, Intel, and Apple Silicon hardware out of the box. What we could not independently verify: any specific throughput benchmark numbers vLLM’s marketing pages cite (e.g. “24x higher throughput”) — we did not reproduce those figures ourselves, so treat them as vendor claims, not measured facts. The functional claims above — the CLI command, the OpenAI-compatible endpoint, the licence — are all confirmed directly against the repo and docs.

---

One open-source tool a week. What it does, what it replaces, and something you can build with it — in under three minutes.

Get next week’s tool by email — sign-up coming shortly.