Skip to content

Training

Fine-tune Qwen3-8B with LoRA: a step-by-step guide

Data format, thinking mode, a complete PEFT + TRL configuration, how many steps you're running, reading train and eval loss, proving the adapter helps, and the same run in Kojen.

By Will Schulz10 min read
A faceted obsidian monolith with a thin glowing red glass plate clipped to its face, traced with circuit-like lines

Qwen3-8B is one of the most popular open-weights bases to fine-tune: Apache-2.0 licensed, 8.2B parameters, a 32K-token native context, and good tool use for its size.[1] This guide walks through a LoRA fine-tune end to end: deciding whether to do it, shaping the data, the training configuration line by line, reading the loss, and checking that the adapter actually helps. Then it shows the same run in Kojen.

Step 0: make sure you need weights

Fine-tuning is the expensive lever. Before it, measure the base model with a good prompt on a held-out test set, then try automatic prompt optimisation; we compare the options in Prompt tuning or fine-tuning? LoRA earns its place when you need a consistent output format, a house style, a narrow classification or extraction skill, or a smaller model doing what a larger one does with a long prompt. If you need the model to know your documents, use retrieval instead (see fine-tuning vs RAG).

Step 1: shape the data as conversations

Supervised fine-tuning trains the model to produce the assistant turn given the turns before it. Store each example as a list of chat messages, one JSON object per line. TRL's SFTTrainer applies the model's chat template to this format for you.[2]

{"messages": [{"role": "user", "content": "Ticket: I was charged twice for March."}, {"role": "assistant", "content": "billing"}]}
{"messages": [{"role": "user", "content": "Ticket: The export button does nothing."}, {"role": "assistant", "content": "bug"}]}
  • Split three ways before anything else: train, a small eval split for loss during training, and a test split you only use at the end. Deduplicate across splits.
  • Quality beats volume. A few hundred clean, consistent examples of one task usually teach more than thousands of noisy ones. Make the expected outputs exactly what you want in production, including length and format.
  • Keep examples under your max length. Rows longer than max_length are truncated, which silently cuts off the answer the model was meant to learn.

Decide about thinking mode

Qwen3 is a hybrid reasoning model. Thinking is on by default and wraps reasoning in a <think>…</think> block; passing enable_thinking=False to the chat template turns it off.[1] If your training targets are direct answers with no reasoning, serve the fine-tuned model with thinking off too, so the prompt format at inference matches what it was trained on. Whatever you choose, render one example and read it:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
row = {"messages": [{"role": "user", "content": "Ticket: ..."},
                    {"role": "assistant", "content": "billing"}]}
print(tok.apply_chat_template(row["messages"], tokenize=False))
# Look at exactly what the model will be trained on before spending GPU time.

Step 2: configure LoRA

This configuration runs on one GPU with 40 GB or more. It's the same set of hyperparameters Kojen uses by default, so you can compare a local run with a Kojen run.

# pip install "transformers>=4.51" peft trl datasets accelerate
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

base = "Qwen/Qwen3-8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16,
                                             attn_implementation="sdpa")
data = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})

peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)
args = SFTConfig(
    output_dir="qwen3-8b-tickets",
    num_train_epochs=2,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,      # effective batch 16
    gradient_checkpointing=True,
    bf16=True,
    max_length=2048,
    eval_strategy="epoch",
    logging_steps=10,
    seed=42,
)
trainer = SFTTrainer(model=model, args=args, train_dataset=data["train"],
                     eval_dataset=data["eval"], processing_class=tok,
                     peft_config=peft_config)
trainer.train()
trainer.model.save_pretrained("qwen3-8b-tickets/final")   # adapter only, ~87 MB
SettingValueWhy
target_modulesAll 7 projectionsLoRA on attention only learns more slowly; MLP layers matter most
r / lora_alpha16 / 32Enough capacity for one task; alpha = 2r is a common default
learning_rate2e-4LoRA's best rate is about 10× full fine-tuning's
effective batch16LoRA tolerates very large batches less well
epochs1–3More epochs on small data mostly memorise
gradient_checkpointingonTrades about a third more compute for much less memory

The target-module and learning-rate advice comes from Thinking Machines' study of when LoRA matches full fine-tuning,[3] and the adapter mechanics from the original LoRA paper.[4] PEFT's documentation lists the remaining options.[5] If memory is tight, load the base in 4-bit and you have QLoRA; the trade-offs are in LoRA vs QLoRA vs full fine-tuning.

Step 3: train and read the loss

Know how many steps you're running before you start:

1,000 training rows ÷ effective batch 16  = 63 steps per epoch
× 2 epochs                                 = 126 optimiser steps
warmup 3%                                  ≈ 4 steps
logging every 10 steps                     ≈ 13 loss points to look at
  • Training loss should fall quickly, then flatten. A loss that drops to near zero on a small dataset means the model is memorising.
  • Eval loss is the one to watch. If it rises while training loss keeps falling, stop earlier or use fewer epochs.
  • Loss is not accuracy. A lower loss doesn't guarantee better answers on your task, especially when the whole conversation, prompt included, is in the loss.

Step 4: prove it helps

Load the base plus adapter, answer the test split with the same settings you'll use in production, and grade it with the same grader you used for the baseline. Compare the two on the same cases with McNemar's test; with 100 test cases you can reliably detect only large differences, as our sample-size guide explains. Also check a few general prompts outside the task: LoRA forgets less than full fine-tuning, but it can still drift.[6]

To serve the adapter, load it into vLLM next to the base model and call it through an OpenAI-compatible API. That's covered in Serve a LoRA adapter behind an OpenAI-compatible endpoint.

The same run in Kojen

Kojen's Text LoRA runs the configuration above without your own GPU. In the app, go to Pipelines → New pipeline, choose Text LoRA (GPU), pick a dataset with input and expected columns, and choose the base model, epochs, rank and GPU provider. Over MCP or the API it looks like this:

train_model {
  "name": "ticket-router-lora",
  "pipeline_type": "text_lora",
  "project_id": "…",
  "dataset_id": "…",            // input/expected rows, at least 20 for training
  "base_model": "qwen3-8b",     // or "llama-3.1-8b-instruct" (Modal only)
  "epochs": 2,
  "rank": 16,
  "gpu_provider": "cheapest"    // or modal, runpod, vast, lambda
}
  • Kojen exports the dataset's train split as chat-format JSONL for training and its holdout split for eval loss. The test split is never exported, so later evaluations stay clean.
  • Defaults match the code above: rank 16 (max 64), alpha 2 × rank, dropout 0.05, all seven projections, learning rate 2e-4 with cosine decay, effective batch 16, 2 epochs (max 10), 2,048-token max length (max 4,096), bf16 with gradient checkpointing. Up to 5,000 rows by default (max 20,000).
  • gpu_provider: "cheapest" picks the lowest live price among Modal, Runpod, Vast.ai and Lambda with a GPU of at least 44 GB in stock, and falls back if a launch fails. Llama-3.1-8B-Instruct trains on Modal only.
  • The run reserves GPU credits for its longest possible duration up front and is charged only for the GPU time used, including if it fails or is cancelled.
  • It reports real train and eval loss and saves the adapter as a draft model version. Adapters can't serve evaluations or deployments in Kojen yet, so step 4 happens outside Kojen for now.

Frequently asked questions

How do I fine-tune Qwen3-8B with LoRA?
Format examples as chat messages in JSONL, load Qwen/Qwen3-8B in bf16 with transformers 4.51 or later, attach a PEFT LoraConfig (rank 16, alpha 32, all seven projection matrices), and train with TRL's SFTTrainer at a learning rate around 2e-4 for one to three epochs, watching eval loss.
What GPU do I need to fine-tune Qwen3-8B with LoRA?
The bf16 weights take about 16.4 GB and a rank-16 adapter with its optimiser state under 1 GB, so a 40–48 GB GPU is comfortable with gradient checkpointing, and an 80 GB card allows long sequences. QLoRA fits on a 16–24 GB card.
Should I train Qwen3 with thinking mode on or off?
Match training to serving. If your training answers contain no reasoning, serve with enable_thinking=False so the prompt format matches what the model learned. Render an example with the chat template before training to check.
How many examples do I need to fine-tune Qwen3-8B?
There's no fixed number, but a few hundred clean, consistent examples of one task often suffice for format and style. Kojen requires at least 20 training rows and trains on up to 5,000 by default.
What learning rate should I use for LoRA?
Around 1e-4 to 2e-4 is a common starting point. Thinking Machines found LoRA's optimal learning rate is about ten times that of full fine-tuning.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]Qwen3-8B model card. Qwen, Hugging Face.
  2. [2]SFT Trainer. Hugging Face TRL documentation.
  3. [3]LoRA Without Regret. Schulman et al., Thinking Machines Lab (29 September 2025).
  4. [4]LoRA: Low-Rank Adaptation of Large Language Models. Hu et al., arXiv (2021).
  5. [5]LoRA. Hugging Face PEFT documentation.
  6. [6]LoRA Learns Less and Forgets Less. Biderman et al., arXiv (2024).

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →