---
title: "What I learned from training my own model"
author: "Jakub Ćwirlej"
id: training-lessons
language: en
url: "https://jakubcwirlej.com/articles/training-lessons.md"
page: "https://jakubcwirlej.com/ai-dev/"
translations:
  pl: "https://jakubcwirlej.com/pl/articles/training-lessons.md"
  fr: "https://jakubcwirlej.com/fr/articles/training-lessons.md"
  nl: "https://jakubcwirlej.com/nl/articles/training-lessons.md"
updated: 2026-10-14
words: 2503
summary: "Ten practical lessons from the author's experience of training language models himself. They cover optimistic compute estimates, starting small, constant monitoring, isolated changes with ablation tests, data choices, realistic expectations, patience during plateaus and the need for more than one GPU. He closes by arguing that small, specialised models built by individuals still make sense."
summary_type: editorial
---

# What I learned from training my own model

Jakub Ćwirlej · text language: EN · 2503 words

## Summary (editorial)

Ten practical lessons from the author's experience of training language models himself. They cover optimistic compute estimates, starting small, constant monitoring, isolated changes with ablation tests, data choices, realistic expectations, patience during plateaus and the need for more than one GPU. He closes by arguing that small, specialised models built by individuals still make sense.

## Key points (editorial)

- Real hardware utilisation (MFU) is far below what spec sheets suggest; his first model, with 1.5B parameters, ended up undertrained.
- Watching norms and comparing daily outputs reveals problems the loss does not show; he switched from AdamW to MuonH.
- Change one thing at a time; he had to build his dataset from scratch three times.
- Abilities often grow in leaps after long plateaus, and monitoring the top-10 answers can show progress that top-1 hides.
- Small, narrow models that run locally or on a CPU can be more cost-effective than the largest models, and training one is the best way to understand how AI works.

## Full text (the author's own words)

### 1. Back-of-the-envelope calculations will always be too optimistic

When we think about training a model, what naturally comes to mind is the machine-learning process — work done for us by powerful Nvidia clusters in data centres or, if we’re luckier, by our own fiendishly powerful workstation. Reading the spec sheet, it’s easy to forget that real MFU (Model FLOPs Utilization) is much lower than the card makers’ marketing would suggest. Depending on the architecture and the level of detail, the hardware will let you squeeze only a fraction of its compute out of the silicon. What good is it that the card computes in FP4 if it won’t run stably when you’re training a base model from scratch, or if, at your model’s width, going below BF16 gains you nothing?

The moment you realise you can’t train the model you planned in the time you planned, or even at anything like the size you planned, is not an easy one. Especially if you’ve spent weeks getting to the point where you can start training, only to find the whole plan was unrealistic. After all, on paper everything added up! Your first instinct is to reach for optimisations — torch compile, autocast, flash attention, FLCE, heads and minibatches sized to suit CUDA’s preferences. It all works, but you can’t cheat the physics of silicon. If you’re training a small model, its small width will work against you. If you can’t afford to train a large model, you can’t afford to get the most out of your GPU either. Getting to optimal utilisation of the card can take days, and a final drop in training TPS (Tokens Per Second) of 30–40% against your initial calculations means a difference of days, even weeks, of GPU time.

### 2. Always start at the smallest scale that can validate your ideas

My first model had 1.5B parameters, and I only managed to train it to 3× Chinchilla. I made plenty of mistakes building it, but it worked all the same. What left it of so little use was, above all, undertraining. Had I chosen, on the same budget, to train a model ten times smaller, I could have shown off a well-trained, useful model. I’d have had time for testing and would certainly have avoided many mistakes, which would have let me build a better architecture. Of course size matters, but data, architecture and computational efficiency matter even more. The odds of designing a good model from the literature, without tests and without mistakes, are about the same as winning the lottery.

### 3. A model doesn’t train itself

Training a model sounds cool, but it’s far more boring than you might think. In reality it’s mostly a process of constantly designing experiments, taking measurements, analysing the model’s outputs and hunting for solutions to the problems that crop up along the way. All the time, round the clock, day after day. If you can’t take satisfaction from small discoveries and minor wins, it may turn out to be too boring. Yet it is precisely iron discipline in taking measurements that decides everything. Even with well-chosen training parameters, it may turn out over time that your model needs new data, or is degrading in ways that are hard to spot at first glance. You won’t notice without good monitoring and constant testing, and the less often you do them, the more time you’ll lose if the model starts misbehaving.

Take one common scenario. A few days into training, you notice the model is giving shorter and shorter answers while growing more and more sure of itself. The loss and the held-out set look fine, but something is wrong with the model. If you analyse the norms as you go and compare the model’s answers each day with the previous day’s, you can quickly catch the rise in the logits and realise that the model is assigning ever higher probability to particular tokens. This arrogance is a lazy optimiser’s trick: AdamW has found a way to fool us and, instead of teaching the model, is lowering the loss some other way. If you catch it after a few days, you might lose a few dozen dollars; if you just let the run go, counting on the Nvidia card to do all the work, you could spend hundreds or thousands, and the model may already be beyond saving.

I spent weeks myself watching Adam inflate my models’ norms, lowering the effective learning rate (LR), despite QK-norm and a weight decay of 0.1. Training slowed down, driving me to despair, because, as we know, Pythagoras applies here: a growing weight vector gets harder and harder to rotate with a constant LR. The problem is well described in the literature, but until you see it for yourself, you won’t understand why it’s better to use a different optimiser. After all, AdamW has helped train hundreds of models. In the end I went with MuonH, and it was one of the better decisions I’ve made. Of course, a lot of this can be gleaned from the literature, from textbooks, or by following research papers on arXiv, but you never know how it will work in your case. Many solutions that in theory should bring well-documented benefits didn’t work for me. You won’t know until you check for yourself, and to check you have to keep your finger on the pulse and track progress as you go.

### 4. Always change just one thing at a time, and run ablation tests

Over the past few months I’ve tested hundreds of small models — both classic transformers and my own ideas... and today I know I didn’t test enough. Trying to move too fast, I wasted time for nothing. Number of layers, FFN ratio, d_model, weight tying, head size, number of KV heads, batch, optimiser, recurrence, sparsity, training data: dozens of factors affect a model’s behaviour, and their combinations add up to thousands of possible architectures. And although, broadly speaking, everything behaves the way you’d expect after reading your first textbook on AI engineering, until you test your own architecture you won’t know where you stand. Meanwhile, small changes can often lead to big differences in computational efficiency and benchmarks, even with the same dataset and batch size. If you don’t make changes in isolation, it’s easy to lose track. I made this mistake many times myself. It’s not that the tweaks were bad. Though they often were. The real problem was that when I did take a step forward, I couldn’t tell which change I owed it to.

### 5. Start with the data

For training a small model, Hugging Face has an unimaginable wealth of data. You might think that if the data is within arm’s reach, in quantities far beyond the needs of any garage project, the matter is settled. But the choice of data matters far more than it might seem at first. And the fact that it has already been gathered into ready-made datasets doesn’t make things any easier. Then there’s choosing the right tokenizer, the languages you want to teach the model, and the proportions: what percentage of the data should be code, for example, how much maths and how much natural language. Then, which domains your model should specialise in. The further you go, the more options open up. I had to build my dataset from scratch no fewer than three times. I also had a few smaller slip-ups, such as an instruct-template leak I didn’t catch at the data-preparation stage. Each of those mistakes forced me to roll training back by hours or even days.

### 6. Don’t expect too much

Watching the big labs and the smaller ones push the boundaries of what AI can do at every turn, it’s easy to get the impression that training a useful model is basically simple. But there are reasons why pushing those boundaries costs billions of dollars. The fact that your model starts talking like a human, or posts good scores on popular benchmarks, says nothing about its final quality or usefulness. Models learn very fast at the beginning, in the first days of training, but the fight for every further percentage point takes ingenuity, attention and grit. Especially when, for days on end, you watch the model plateau or even go backwards. Besides, popular benchmarks say little about a model’s real capabilities. Your model may be good at Python and score brilliantly on HumanEval and MBPP, but if it doesn’t know about the latest libraries, its real-world usefulness will be modest.

### 7. Be patient

Many of a model’s abilities grow in leaps. This well-documented phenomenon, grokking, is often preceded by a long plateau that can last for days. Watching the model make no progress at all while you pay in dollars for every hour of training, for rented compute or even just for electricity, it’s easy to have a moment of doubt. You think: “The model has hit capacity, it won’t learn any more”, or “The big labs that train to 1000× Chinchilla must be using some tricks”. Yet abilities often grow in leaps. All it takes is for the model to learn to close off some function, and suddenly, instead of one MBPP task, a dozen or so pass. It’s also often the case that the model is making real progress, but if, to save money, you only monitor top-1 answers instead of top-10, you may not notice it at all for a long time. The model has to learn to assign higher confidence, and that takes time.

### 8. One card isn’t enough

It may seem that if you have an H100 or an RTX PRO 6000 (96 GB) at your disposal, the only thing limiting you is your budget. The problem is that training requires constant tests and measurements to tell you what stage you’re at. You may also need an extra GPU to generate more training data. I myself made significant progress training my model with the STaR method (in which my own model generates a large number of tasks and the correct ones are added to the dataset) and with a dataset of examples of translations between programming languages

Pausing training on a powerful card while the model generates answers is very costly and simply not worth it, and generating alongside training — even if the VRAM allows it — can be seriously risky. The last thing you want is to find out three days later, when the model’s results drop, that while you were generating on the card alongside the training run there was, say, a momentary “starvation” of the data loader, after which the loss quietly stopped falling. At a later stage of training, things like this are often extremely hard to catch, even if you have monitoring that warns you of norm spikes.

### 9. Start with something simple

I started building my first model on a hunch that an LLM’s own activations could be used in training, and that tracking them as it learned should help build a smarter, more truthful model. Today I can’t get over how naive that approach was. Not because I now think it’s impossible, but because, with no idea what LLM training actually looks like, I started experimenting with something that was beyond me. I had to absorb an enormous amount of knowledge while testing both solutions from classic transformer architecture and my own ideas, which — as time showed — were largely impossible to implement.

Along the way I ran into a whole series of problems: once normalised, the SwiGLU gate activation signal lost its meaning, and without normalisation AdamW used my modifications to inflate the model’s norms. Only later did I understand that many of the advantages I’d found in my first model came from something resembling noise regularisation. On top of that, just computing the Gini of the SwiGLU gate activations was computationally expensive, adding a few percent of extra compute that the results didn’t repay in quality.

I’m currently training a small 160M model with a standard architecture that specialises in coding. It was the best decision I could have made, because I gave myself a chance to create something that could be genuinely useful. And the model’s first results show that learning from my mistakes hasn’t gone to waste.

### 10. Don’t build a model that’s good at everything

Given the astronomical costs, and whether it can even be done, building a general-purpose model as an individual or a small organisation makes no sense whatsoever today, for purely economic reasons. Competing with the giants, or even with the smaller labs releasing open models, is a battle you can’t win without huge investment and experience. But that doesn’t mean building your own models makes no sense.

You can imagine hundreds of applications where small models, trained from scratch or fine-tuned on unique data, will have an edge over the giants — and that edge needn’t lie in knowledge. It’s enough for the model to be fast and able to run locally in a browser, or on the CPU of any workstation. For specific, narrow uses, a micro model of a few hundred million parameters may be far more cost-effective than harnessing the most powerful models available from OpenAI or Anthropic. It’s not just about cost. Though the difference is colossal. A model that specialises in a narrow area is not only much easier to train, but also easier to control and adapt to the user’s needs. Most importantly, though, it’s a task that isn’t beyond an ordinary person who’s ready to take on the challenge.

The big labs’ models are trained on thousands of very expensive chips. To take one example, training a Llama 3 400B-class model required a cluster of 16,000 H100s. The cost runs into tens of millions of dollars. Training the latest GPT Astra has probably raised the bar much higher. Given that the whole sector is growing on credit, and that rolling out AI at scale, with current API costs and data-security concerns, is often hard to justify as a business case, in the longer term we can expect small models to play an ever bigger role. Small models will also keep getting more efficient. I wouldn’t be surprised if, in future, a model of around 7B parameters were enough for most of our tasks. The bursting of the “AI bubble”, which the media are talking about more and more, may only speed this process up.

That’s why I believe building small models makes a great deal of sense. First, there’s no better way to understand how AI works than to train your own model. Second, although in the big labs AI development is increasingly in the hands of... AI itself, we can expect that humanity, taking safety seriously, will demand control and human oversight of the training process. And that requires understanding how the process works in practice. Of course, go on LinkedIn today and you’re flooded with posts about AI from all sorts of specialists offering tutorials and courses. But how many of them have actually trained their own model?

---

Jakub Ćwirlej — Journalist who builds with AI · on the site: https://jakubcwirlej.com/ai-dev/
Full context: https://jakubcwirlej.com/context/en.md · https://jakubcwirlej.com/llms.txt