Skip to content

Shipping

Serve a LoRA adapter behind an OpenAI-compatible endpoint

Merge an adapter or serve it on a shared base, a working vLLM setup for Qwen3-8B with LoRA, how many conversations fit on a 48 GB GPU, and what a production endpoint needs beyond the server.

By Will Schulz9 min read
A dark metal server block with thin glass cartridges slotted into its front, one glowing red, sending a beam of red light to the right

A LoRA fine-tune gives you an adapter: a folder with an adapter_config.json and a weights file of tens to hundreds of megabytes, not a model. To use it, something has to load the base model, apply the adapter and answer requests. Doing that behind an OpenAI-compatible API means every client, SDK and agent framework that already speaks OpenAI's chat format can call your fine-tune with only the base URL and model name changed.

This guide covers the two ways to serve an adapter, a working vLLM setup for Qwen3-8B, how many adapters and conversations fit on one GPU, and what a production endpoint needs beyond the server process.

Merge it, or serve it alongside the base?

The LoRA update is a low-rank matrix BA added to a frozen weight W. You can either fold it in permanently, W′ = W + BA, which adds no inference latency, as the LoRA paper pointed out,[1] or keep the base and apply the adapter per request.

Merge into the baseServe adapters on a shared base
What you deployA full 16 GB model per fine-tuneOne base plus ~87 MB per adapter
LatencySame as the base modelSmall overhead for the adapter matmuls
Many fine-tunesOne GPU (or more) eachMany adapters on one GPU
Swapping versionsRedeploy the modelLoad a new adapter by name
Works withAny inference serverServers with multi-LoRA support

If you have one fine-tune and high traffic, merge. If you have several (per customer, per task, per version under test), serve them on a shared base. The systems research is clear that batching requests for different adapters together is efficient: S-LoRA reported up to four times the throughput of naive serving and thousands of adapters on one GPU,[2] and Punica reported 12 times the throughput of the serving systems it compared against, at about 2 ms extra latency per token.[3]

Merging with PEFT

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "./adapters/tickets").merge_and_unload()
merged.save_pretrained("./qwen3-8b-tickets")              # a full 16 GB model
AutoTokenizer.from_pretrained("Qwen/Qwen3-8B").save_pretrained("./qwen3-8b-tickets")

PEFT's merge_and_unload returns an ordinary Transformers model,[4] which any OpenAI-compatible server can serve with vllm serve ./qwen3-8b-tickets. Merge into the same base, in 16-bit, that you trained against; merging into a quantised copy changes the result.

Serving adapters with vLLM

vLLM's OpenAI-compatible server supports LoRA adapters natively. Start it with --enable-lora and register adapters with --lora-modules; the JSON form records the base model each adapter belongs to.[5]

pip install vllm

vllm serve Qwen/Qwen3-8B \
  --enable-lora \
  --max-lora-rank 64 \
  --lora-modules '{"name": "tickets", "path": "./adapters/tickets", "base_model_name": "Qwen/Qwen3-8B"}' \
  --enable-auto-tool-choice --tool-call-parser hermes \
  --reasoning-parser qwen3 \
  --api-key "$VLLM_API_KEY"
  • --max-lora-rank must be at least your largest adapter's rank. Its default is 16, so a rank-32 or rank-64 adapter fails to load without it. --max-loras (default 1) is how many different adapters can run in the same batch.[6]
  • The tool-calling and reasoning flags follow Qwen's own vLLM instructions for Qwen3;[7] other base models use other parsers.
  • --api-key makes vLLM require a bearer token. Without it, anyone who can reach the port can use your GPU.

Check what the server exposes. Each adapter is listed as its own model:

curl -s localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
# → "Qwen/Qwen3-8B" (the base) and "tickets" (parent: "Qwen/Qwen3-8B")

Then call it like any OpenAI model. The adapter is selected by the model field:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key=VLLM_API_KEY)
reply = client.chat.completions.create(
    model="tickets",                               # the adapter; use the base name for no adapter
    messages=[{"role": "user", "content": "Ticket: I was charged twice for March."}],
    temperature=0.7,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},  # match training
)
print(reply.choices[0].message.content)

If you fine-tuned on direct answers without reasoning, turn Qwen3's thinking mode off at inference too, so the prompt matches what the adapter was trained on.[8]

Loading adapters without restarting

With VLLM_ALLOW_RUNTIME_LORA_UPDATING=True, vLLM adds /v1/load_lora_adapter and /v1/unload_lora_adapter, so you can register a new version by name while the server runs. vLLM's documentation warns this is a security risk to enable only in isolated, fully trusted environments,[5] so keep those routes off the public internet.

How much fits on one GPU

Weights are fixed; what limits concurrency is the KV cache, the memory for each in-flight conversation's attention keys and values. For Qwen3-8B's architecture (36 layers, 8 KV heads of 128 dimensions),[8] our estimate for a 48 GB card such as an L40S:

Qwen3-8B KV cache per token
= 2 (K and V) × 36 layers × 8 KV heads × 128 dims × 2 bytes = 147,456 bytes ≈ 144 KB

48 GB GPU at vLLM's default 90% memory use      = 43.2 GB
− bf16 weights (16.4 GB) − ~2 GB runtime         ≈ 24.8 GB for KV cache
24.8 GB ÷ 144 KB                                 ≈ 168,000 tokens
                                                 ≈ 41 concurrent 4,096-token conversations
Each rank-16 adapter adds ~87 MB.

An 80 GB card roughly doubles the conversation count. Adapters themselves are cheap to keep resident; the real cost of serving many fine-tunes is that requests for different adapters share one batch, so the throughput of the busiest one sets the pace.

What a production endpoint needs

A bare vLLM process is a model server, not an endpoint. Before real users call it, add:

  • Keys per client, not one shared token, so you can revoke one without breaking the others.
  • Rate limits per key and per adapter, so one caller can't starve the rest.
  • Request logging with latency, token counts and errors per adapter, and a policy for what you store, since prompts contain user data.
  • A way to collect feedback on individual responses, which becomes your next training set.
  • A rollback path: keep the previous adapter loaded under its own name and switch clients back by changing the model name.
  • Evaluation before promotion: run the new adapter and the current one on the same test set and compare them with a paired test. See how to evaluate AI agents.

For the training side, see Fine-tune Qwen3-8B with LoRA; for the loop after launch, see Shipping is where training starts.

Frequently asked questions

How do I serve a LoRA adapter with an OpenAI-compatible API?
Run vLLM with --enable-lora and register the adapter with --lora-modules name=path (or the JSON form with base_model_name). The adapter appears in /v1/models and is selected by setting the model field of a chat completions request to its name.
Should I merge my LoRA adapter or serve it separately?
Merge when you have one fine-tune and want base-model latency on any server. Serve adapters on a shared base when you have several fine-tunes or versions: a rank-16 adapter for an 8B model is under 100 MB instead of a 16 GB model copy.
Why does vLLM fail to load my LoRA adapter?
A common cause is rank: --max-lora-rank defaults to 16, so adapters trained at rank 32 or 64 need it raised. The adapter must also match the base model it was trained on.
How many LoRA adapters can one GPU serve?
Many: each rank-16 adapter for an 8B model is about 87 MB, and S-LoRA served thousands on one GPU. Concurrency is limited by KV cache instead; for Qwen3-8B on a 48 GB card we estimate about 168,000 cached tokens, or about 41 concurrent 4,096-token conversations.
Can Kojen deploy a LoRA adapter?
Not yet. Kojen trains real text LoRA adapters on Qwen3-8B and Llama-3.1-8B-Instruct and saves them as draft model versions, but its OpenAI-compatible deployments currently serve hosted models and prompt-tuned versions, not adapters.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]LoRA: Low-Rank Adaptation of Large Language Models. Hu et al., arXiv (2021).
  2. [2]S-LoRA: Serving Thousands of Concurrent LoRA Adapters. Sheng et al., arXiv (2023).
  3. [3]Punica: Multi-Tenant LoRA Serving. Chen et al., arXiv (2023).
  4. [4]LoRA: merge adapters. Hugging Face PEFT documentation.
  5. [5]LoRA Adapters. vLLM documentation.
  6. [6]Engine Arguments. vLLM documentation.
  7. [7]vLLM deployment. Qwen documentation.
  8. [8]Qwen3-8B model card and config. Qwen, Hugging Face.

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →