A LoRA fine-tune gives you an adapter: a folder with an adapter_config.json and a weights file of tens to hundreds of megabytes, not a model. To use it, something has to load the base model, apply the adapter and answer requests. Doing that behind an OpenAI-compatible API means every client, SDK and agent framework that already speaks OpenAI's chat format can call your fine-tune with only the base URL and model name changed.
This guide covers the two ways to serve an adapter, a working vLLM setup for Qwen3-8B, how many adapters and conversations fit on one GPU, and what a production endpoint needs beyond the server process.
Merge it, or serve it alongside the base?
The LoRA update is a low-rank matrix BA added to a frozen weight W. You can either fold it in permanently, W′ = W + BA, which adds no inference latency, as the LoRA paper pointed out,[1] or keep the base and apply the adapter per request.
| Merge into the base | Serve adapters on a shared base | |
|---|---|---|
| What you deploy | A full 16 GB model per fine-tune | One base plus ~87 MB per adapter |
| Latency | Same as the base model | Small overhead for the adapter matmuls |
| Many fine-tunes | One GPU (or more) each | Many adapters on one GPU |
| Swapping versions | Redeploy the model | Load a new adapter by name |
| Works with | Any inference server | Servers with multi-LoRA support |
If you have one fine-tune and high traffic, merge. If you have several (per customer, per task, per version under test), serve them on a shared base. The systems research is clear that batching requests for different adapters together is efficient: S-LoRA reported up to four times the throughput of naive serving and thousands of adapters on one GPU,[2] and Punica reported 12 times the throughput of the serving systems it compared against, at about 2 ms extra latency per token.[3]
Merging with PEFT
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "./adapters/tickets").merge_and_unload()
merged.save_pretrained("./qwen3-8b-tickets") # a full 16 GB model
AutoTokenizer.from_pretrained("Qwen/Qwen3-8B").save_pretrained("./qwen3-8b-tickets")PEFT's merge_and_unload returns an ordinary Transformers model,[4] which any OpenAI-compatible server can serve with vllm serve ./qwen3-8b-tickets. Merge into the same base, in 16-bit, that you trained against; merging into a quantised copy changes the result.
Serving adapters with vLLM
vLLM's OpenAI-compatible server supports LoRA adapters natively. Start it with --enable-lora and register adapters with --lora-modules; the JSON form records the base model each adapter belongs to.[5]
pip install vllm
vllm serve Qwen/Qwen3-8B \
--enable-lora \
--max-lora-rank 64 \
--lora-modules '{"name": "tickets", "path": "./adapters/tickets", "base_model_name": "Qwen/Qwen3-8B"}' \
--enable-auto-tool-choice --tool-call-parser hermes \
--reasoning-parser qwen3 \
--api-key "$VLLM_API_KEY"--max-lora-rankmust be at least your largest adapter's rank. Its default is 16, so a rank-32 or rank-64 adapter fails to load without it.--max-loras(default 1) is how many different adapters can run in the same batch.[6]- The tool-calling and reasoning flags follow Qwen's own vLLM instructions for Qwen3;[7] other base models use other parsers.
--api-keymakes vLLM require a bearer token. Without it, anyone who can reach the port can use your GPU.
Check what the server exposes. Each adapter is listed as its own model:
curl -s localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY" # → "Qwen/Qwen3-8B" (the base) and "tickets" (parent: "Qwen/Qwen3-8B")
Then call it like any OpenAI model. The adapter is selected by the model field:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key=VLLM_API_KEY)
reply = client.chat.completions.create(
model="tickets", # the adapter; use the base name for no adapter
messages=[{"role": "user", "content": "Ticket: I was charged twice for March."}],
temperature=0.7,
extra_body={"chat_template_kwargs": {"enable_thinking": False}}, # match training
)
print(reply.choices[0].message.content)If you fine-tuned on direct answers without reasoning, turn Qwen3's thinking mode off at inference too, so the prompt matches what the adapter was trained on.[8]
Loading adapters without restarting
With VLLM_ALLOW_RUNTIME_LORA_UPDATING=True, vLLM adds /v1/load_lora_adapter and /v1/unload_lora_adapter, so you can register a new version by name while the server runs. vLLM's documentation warns this is a security risk to enable only in isolated, fully trusted environments,[5] so keep those routes off the public internet.
How much fits on one GPU
Weights are fixed; what limits concurrency is the KV cache, the memory for each in-flight conversation's attention keys and values. For Qwen3-8B's architecture (36 layers, 8 KV heads of 128 dimensions),[8] our estimate for a 48 GB card such as an L40S:
Qwen3-8B KV cache per token
= 2 (K and V) × 36 layers × 8 KV heads × 128 dims × 2 bytes = 147,456 bytes ≈ 144 KB
48 GB GPU at vLLM's default 90% memory use = 43.2 GB
− bf16 weights (16.4 GB) − ~2 GB runtime ≈ 24.8 GB for KV cache
24.8 GB ÷ 144 KB ≈ 168,000 tokens
≈ 41 concurrent 4,096-token conversations
Each rank-16 adapter adds ~87 MB.An 80 GB card roughly doubles the conversation count. Adapters themselves are cheap to keep resident; the real cost of serving many fine-tunes is that requests for different adapters share one batch, so the throughput of the busiest one sets the pace.
What a production endpoint needs
A bare vLLM process is a model server, not an endpoint. Before real users call it, add:
- Keys per client, not one shared token, so you can revoke one without breaking the others.
- Rate limits per key and per adapter, so one caller can't starve the rest.
- Request logging with latency, token counts and errors per adapter, and a policy for what you store, since prompts contain user data.
- A way to collect feedback on individual responses, which becomes your next training set.
- A rollback path: keep the previous adapter loaded under its own name and switch clients back by changing the model name.
- Evaluation before promotion: run the new adapter and the current one on the same test set and compare them with a paired test. See how to evaluate AI agents.
For the training side, see Fine-tune Qwen3-8B with LoRA; for the loop after launch, see Shipping is where training starts.



