Qwen3-8B is one of the most popular open-weights bases to fine-tune: Apache-2.0 licensed, 8.2B parameters, a 32K-token native context, and good tool use for its size.[1] This guide walks through a LoRA fine-tune end to end: deciding whether to do it, shaping the data, the training configuration line by line, reading the loss, and checking that the adapter actually helps. Then it shows the same run in Kojen.
Step 0: make sure you need weights
Fine-tuning is the expensive lever. Before it, measure the base model with a good prompt on a held-out test set, then try automatic prompt optimisation; we compare the options in Prompt tuning or fine-tuning? LoRA earns its place when you need a consistent output format, a house style, a narrow classification or extraction skill, or a smaller model doing what a larger one does with a long prompt. If you need the model to know your documents, use retrieval instead (see fine-tuning vs RAG).
Step 1: shape the data as conversations
Supervised fine-tuning trains the model to produce the assistant turn given the turns before it. Store each example as a list of chat messages, one JSON object per line. TRL's SFTTrainer applies the model's chat template to this format for you.[2]
{"messages": [{"role": "user", "content": "Ticket: I was charged twice for March."}, {"role": "assistant", "content": "billing"}]}
{"messages": [{"role": "user", "content": "Ticket: The export button does nothing."}, {"role": "assistant", "content": "bug"}]}- Split three ways before anything else: train, a small eval split for loss during training, and a test split you only use at the end. Deduplicate across splits.
- Quality beats volume. A few hundred clean, consistent examples of one task usually teach more than thousands of noisy ones. Make the expected outputs exactly what you want in production, including length and format.
- Keep examples under your max length. Rows longer than
max_lengthare truncated, which silently cuts off the answer the model was meant to learn.
Decide about thinking mode
Qwen3 is a hybrid reasoning model. Thinking is on by default and wraps reasoning in a <think>…</think> block; passing enable_thinking=False to the chat template turns it off.[1] If your training targets are direct answers with no reasoning, serve the fine-tuned model with thinking off too, so the prompt format at inference matches what it was trained on. Whatever you choose, render one example and read it:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
row = {"messages": [{"role": "user", "content": "Ticket: ..."},
{"role": "assistant", "content": "billing"}]}
print(tok.apply_chat_template(row["messages"], tokenize=False))
# Look at exactly what the model will be trained on before spending GPU time.Step 2: configure LoRA
This configuration runs on one GPU with 40 GB or more. It's the same set of hyperparameters Kojen uses by default, so you can compare a local run with a Kojen run.
# pip install "transformers>=4.51" peft trl datasets accelerate
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer
base = "Qwen/Qwen3-8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16,
attn_implementation="sdpa")
data = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})
peft_config = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
args = SFTConfig(
output_dir="qwen3-8b-tickets",
num_train_epochs=2,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # effective batch 16
gradient_checkpointing=True,
bf16=True,
max_length=2048,
eval_strategy="epoch",
logging_steps=10,
seed=42,
)
trainer = SFTTrainer(model=model, args=args, train_dataset=data["train"],
eval_dataset=data["eval"], processing_class=tok,
peft_config=peft_config)
trainer.train()
trainer.model.save_pretrained("qwen3-8b-tickets/final") # adapter only, ~87 MB| Setting | Value | Why |
|---|---|---|
| target_modules | All 7 projections | LoRA on attention only learns more slowly; MLP layers matter most |
| r / lora_alpha | 16 / 32 | Enough capacity for one task; alpha = 2r is a common default |
| learning_rate | 2e-4 | LoRA's best rate is about 10× full fine-tuning's |
| effective batch | 16 | LoRA tolerates very large batches less well |
| epochs | 1–3 | More epochs on small data mostly memorise |
| gradient_checkpointing | on | Trades about a third more compute for much less memory |
The target-module and learning-rate advice comes from Thinking Machines' study of when LoRA matches full fine-tuning,[3] and the adapter mechanics from the original LoRA paper.[4] PEFT's documentation lists the remaining options.[5] If memory is tight, load the base in 4-bit and you have QLoRA; the trade-offs are in LoRA vs QLoRA vs full fine-tuning.
Step 3: train and read the loss
Know how many steps you're running before you start:
1,000 training rows ÷ effective batch 16 = 63 steps per epoch × 2 epochs = 126 optimiser steps warmup 3% ≈ 4 steps logging every 10 steps ≈ 13 loss points to look at
- Training loss should fall quickly, then flatten. A loss that drops to near zero on a small dataset means the model is memorising.
- Eval loss is the one to watch. If it rises while training loss keeps falling, stop earlier or use fewer epochs.
- Loss is not accuracy. A lower loss doesn't guarantee better answers on your task, especially when the whole conversation, prompt included, is in the loss.
Step 4: prove it helps
Load the base plus adapter, answer the test split with the same settings you'll use in production, and grade it with the same grader you used for the baseline. Compare the two on the same cases with McNemar's test; with 100 test cases you can reliably detect only large differences, as our sample-size guide explains. Also check a few general prompts outside the task: LoRA forgets less than full fine-tuning, but it can still drift.[6]
To serve the adapter, load it into vLLM next to the base model and call it through an OpenAI-compatible API. That's covered in Serve a LoRA adapter behind an OpenAI-compatible endpoint.
The same run in Kojen
Kojen's Text LoRA runs the configuration above without your own GPU. In the app, go to Pipelines → New pipeline, choose Text LoRA (GPU), pick a dataset with input and expected columns, and choose the base model, epochs, rank and GPU provider. Over MCP or the API it looks like this:
train_model {
"name": "ticket-router-lora",
"pipeline_type": "text_lora",
"project_id": "…",
"dataset_id": "…", // input/expected rows, at least 20 for training
"base_model": "qwen3-8b", // or "llama-3.1-8b-instruct" (Modal only)
"epochs": 2,
"rank": 16,
"gpu_provider": "cheapest" // or modal, runpod, vast, lambda
}- Kojen exports the dataset's train split as chat-format JSONL for training and its holdout split for eval loss. The test split is never exported, so later evaluations stay clean.
- Defaults match the code above: rank 16 (max 64), alpha 2 × rank, dropout 0.05, all seven projections, learning rate 2e-4 with cosine decay, effective batch 16, 2 epochs (max 10), 2,048-token max length (max 4,096), bf16 with gradient checkpointing. Up to 5,000 rows by default (max 20,000).
gpu_provider: "cheapest"picks the lowest live price among Modal, Runpod, Vast.ai and Lambda with a GPU of at least 44 GB in stock, and falls back if a launch fails. Llama-3.1-8B-Instruct trains on Modal only.- The run reserves GPU credits for its longest possible duration up front and is charged only for the GPU time used, including if it fails or is cancelled.
- It reports real train and eval loss and saves the adapter as a draft model version. Adapters can't serve evaluations or deployments in Kojen yet, so step 4 happens outside Kojen for now.



