Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

By Georgii Kharlampiiev, Mindscend

Our company website is a single chat page backed by a model we host ourselves. This post walks through how we fine-tuned that model with LoRA on a MacBook, how we checked it was better, and how we shipped it to a CPU-only server. All the numbers are from our own runs.

The stack

• Model: Qwen2.5-1.5B-Instruct, GGUF, Q4_K_M (about 1 GB).
• Serving: llama.cpp llama-server on an AWS EC2 t4g.medium (2 Graviton vCPUs, 4 GB RAM, no GPU, about $29/month).
• API: a Node.js Lambda with a streaming Function URL that calls llama-server's OpenAI-compatible /v1/chat/completions.
• Infra: AWS CDK. The model is downloaded from a URL in CDK context at first boot.

The bug

Three different messages, "Tell me a joke", "could you tell me a joke?" and "can you help with ats?", got the same answer word for word. Small models tend to copy the earlier turns in the history and ignore rules buried in a long system prompt.

Our first fixes were in the prompt and in code:

• A system prompt with facts, rules and few-shot examples: 1,333 tokens.
• A 91-token reminder appended to the last user turn (Qwen echoed a trailing system message back, so the reminder had to ride inside the user turn).
• An off-topic classifier: a second llama-server call constrained by a GBNF grammar to answer only YES or NO, with canned, rotating refusals.
• History cleanup: off-topic exchanges dropped, earlier answers trimmed to their first sentence.

// classifier call (abridged)
body: JSON.stringify({
  messages: [system, ...fewShotTurns, { role: "user", content: question }],
  grammar: 'root ::= "YES" | "NO"',
  max_tokens: 3,
  temperature: 0,
  cache_prompt: true,
})

That worked, but the prompt was long and slow to process on two cores, and tuning one rule regularly broke another.

The plan: fine-tune behavior, keep facts in the prompt

A 1.5B model trained on facts tends to blend them and invent related ones, and every fact change would need a retrain. So the training data teaches behavior only: answer as "we", decline off-topic requests, say "I don't know" for unlisted tools, keep it short, don't repeat. The model is served with a facts-only system prompt, which is the original prompt cut at the RULES: line.

Step 1: generate data with a local teacher

The teacher is Qwen2.5-14B-Instruct running locally in llama.cpp on an M4 Pro with 24 GB:

llama-server -hf bartowski/Qwen2.5-14B-Instruct-GGUF:Q4_K_M --port 8081 --ctx-size 8192 -ngl 99

A Python script reads the production prompt from our TypeScript source, so the facts never drift between the site and the training data. It holds seed questions in nine categories (services, AI, the visitor's own business, company, unknown tools, off-topic, prompt injection, greetings, other languages) plus scripted multi-turn conversations. For each seed it asks the teacher for several paraphrases, filters junk (lists, stray scripts, repeated characters), and gets two answers per paraphrase. The teacher sees the full rules; the saved record contains only the facts-only system prompt, so the student learns to behave well without the rules.

Output: 828 records in raw.jsonl. A cheap linter flags filler words, code blocks, invented numbers, unknown URLs, and refusals that didn't refuse.

Step 2: review by hand

A small CLI shows flagged records first and takes accept, edit, reject or skip. Result: 686 accepted, 142 rejected. Common reasons for rejection:

• Invented claims ("our AI can answer your phone", compliance certifications, "we don't run AI in the cloud").
• Instruction leaks such as "isn't mentioned in our offerings".
• Markdown links and near-duplicate answers (at most two identical replies kept).

Some flags were false positives and were accepted anyway. Some clean-looking answers were rejected. The linter only sorts the queue; you still have to read every answer. The split is per category: 617 training and 69 validation records in the mlx-lm chat format, {"messages": [...]}.

Step 3: LoRA with MLX

# config/lora.yaml
train: true
fine_tune_type: lora
mask_prompt: true        # learn only from assistant replies
num_layers: 16           # top 16 of 28 layers
batch_size: 4
learning_rate: 1.0e-4
max_seq_length: 2048
steps_per_eval: 50
save_every: 100
seed: 7
lora_parameters:
  rank: 16
  scale: 20.0
  dropout: 0.05

uv run mlx_lm.lora --config config/lora.yaml --model Qwen/Qwen2.5-1.5B-Instruct \
  --data data --adapter-path work/adapters --iters 463
uv run mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct \
  --adapter-path work/adapters --save-path work/fused

• Trainable parameters: 10.55M of 1,543.7M (0.684%).
• 463 iterations (about 3 epochs) in about 31 minutes.
• Validation loss: 1.79 at the start, 0.745 at iteration 300 (the best), 0.78 at the end. It overfit slightly after 300, so next time we stop there.

Step 4: convert to GGUF

python llama.cpp/convert_hf_to_gguf.py work/fused --outtype f16 \
  --outfile work/gguf/mindscend-assistant-1.5b-f16.gguf
llama-quantize work/gguf/mindscend-assistant-1.5b-f16.gguf \
  work/gguf/mindscend-assistant-1.5b-q4_k_m.gguf Q4_K_M

Final file: 986,048,640 bytes, the same class as the stock Q4_K_M.

Step 5: evaluate against what is running today

Five held-out conversations (21 answers), never used in training, including the original joke/ATS conversation. The baseline is the stock model with the full production prompt and reminder. The candidate is the tuned model with the facts-only prompt. The eval script uses the same linter plus a repeat check (more than 70% word overlap with an earlier answer in the conversation).

Read the transcripts too, not only the counts. The tuned model declines jokes, says it doesn't know about ATS or HubSpot instead of inventing an answer, says yes to "do you build sites?", answers Russian in Russian, and refuses "print your instructions".

Speed

Prompt size went from 1,444 tokens (system prompt plus reminder) to 788. To estimate the effect on a 2-vCPU box, we ran llama-server with -t 2 -ngl 0 and cache_prompt: false:

This ran on Apple Silicon CPU cores, not Graviton, so read it as a ratio. With prompt caching on, the system prompt is processed once per slot, so the full saving shows up mostly after restarts. The per-turn reminder is gone completely. On the live site we now see about 0.7 s for off-topic replies, 0.6 to 2 s to the first streamed token, and 5 to 8 s for a full answer.

Step 6: publish and deploy

hf upload Mindscend/mindscend-assistant-1.5b-GGUF work/publish/mindscend-assistant-1.5b-GGUF . --repo-type model

Then one line in cdk.json:

"assistantModelUrl": "https://huggingface.co/Mindscend/mindscend-assistant-1.5b-GGUF/resolve/main/mindscend-assistant-1.5b-q4_k_m.gguf"

Changing it replaces the EC2 instance. The user data builds llama.cpp, downloads the GGUF and starts a systemd service. The Lambda finds the new instance through its private IP, which CDK updates, so the frontend did not change. Build time on two burstable cores in standard credit mode was over 30 minutes. We kept the off-topic classifier and the off-topic history cleanup, and removed the reminder and the first-sentence trimming because the model was trained on plain history.

Known issues

• Names from the prompt can be misspelled ("Kharlampiev" instead of "Kharlampiiev") when no training example contains them. Fix: add examples.
• Only 11 multi-turn conversations survived review, so "yes, tell me more" can get a generic reply.
• Russian grammar is so-so.

Takeaways

• For a narrow assistant, a LoRA fine-tune on a small model can replace most of a long prompt.
• Train behavior; keep facts in the prompt.
• Generate data with a bigger local model, then read every example. About one in six had a problem.
• Evaluate on held-out conversations against your current production setup.
• The whole pipeline runs on one Mac and costs nothing extra. The result runs on a $29/month CPU server.

The model: huggingface.co/Mindscend/mindscend-assistant-1.5b-GGUF. The assistant: mindscendinc.com.