DEV Community

Cover image for Catastrophic forgetting: why your fine-tuned model got worse at things you didn't train it on
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Catastrophic forgetting: why your fine-tuned model got worse at things you didn't train it on

#finetuning #catastrophicforgetting #evaluation

A student fine-tuned a model hard on customer-support replies and it got noticeably worse at basic reasoning questions it had handled fine before. That's catastrophic forgetting — narrow, aggressive fine-tuning overwrites general capability the base model had.

Why it happens

Too many epochs on too narrow a dataset pushes the model's weights hard in one direction. LoRA reduces this risk versus full fine-tuning because it touches far fewer parameters, but it isn't immune, especially at high rank and learning rate.

How to prevent it

  • Evaluate on a general-capability set before and after fine-tuning, not just on the target task.
  • Fewer epochs and a lower learning rate first; add more only if the target task genuinely needs it.
  • Mix a small slice of general instruction data into the fine-tuning set alongside the task-specific data.

See reading a loss curve during fine-tuning for how forgetting often shows up as a training loss that looks fine while eval quality quietly drops elsewhere.

About Pranjul Rathour

Pranjul Rathour holding a microphone while answering a question during a session
Taking questions during a session

Pranjul Rathour in a grey suit at Dr. Virendra Swarup Institute of Computer Studies (VSICS), Kanpur
At VSICS, Kanpur

Pranjul Rathour in front of an Integral Startup Foundation hackathon backdrop
At an Integral Startup Foundation hackathon

Pranjul Rathour in a checked shirt inside a packed college auditorium
In a packed college auditorium

Pranjul Rathour in a suit and tie with a lanyard at a formal campus event
At a formal campus event

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: [email protected] · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)