--- title: "How I got started" author: "Jakub Ćwirlej" id: how-i-started language: en url: "https://jakubcwirlej.com/articles/how-i-started.md" page: "https://jakubcwirlej.com/ai-dev/" translations: pl: "https://jakubcwirlej.com/pl/articles/how-i-started.md" fr: "https://jakubcwirlej.com/fr/articles/how-i-started.md" nl: "https://jakubcwirlej.com/nl/articles/how-i-started.md" updated: 2026-10-14 words: 1518 summary: "The author recounts how building OrsonAI, a set of AI tools for his own journalistic work, led him from vibe coding to research on language-model hallucinations and to training his own models. He describes the industry's cool reception of the app, his studies of SwiGLU gate activations and the failure of his first model built on his own architecture. Today he trains EUGEN 160M, a small model specialised in code that can be tried in a terminal on the site." summary_type: editorial --- # How I got started Jakub Ćwirlej · text language: EN · 1518 words ## Summary (editorial) The author recounts how building OrsonAI, a set of AI tools for his own journalistic work, led him from vibe coding to research on language-model hallucinations and to training his own models. He describes the industry's cool reception of the app, his studies of SwiGLU gate activations and the failure of his first model built on his own architecture. Today he trains EUGEN 160M, a small model specialised in code that can be tried in a terminal on the site. ## Key points (editorial) - OrsonAI offered live transcription of interviews, an editing mode for consulting a chatbot about selected passages, and certificates of human authorship recorded on the Hedera Hashgraph public ledger. - He now describes the GSI signals he found as knowledge proxy signals rather than a model's “epistemic awareness”, and observed that confabulation is largely a two-stage process. - EUGEN 160M's progress rests on close monitoring, targeted training data, STaR and LEMA synthetic data and examples of translation from Python to C++, Java and Rust. - With the help of an AI coding agent he built his own Triton kernel for training a sparse model: in a ~300M model the sparse variant trained 1.23× faster than the dense one, and later kernel optimisations raised throughput by a further 36%. - He attributes the failure of his first model to a philosophical problem, citing Tarski's definition of truth and Gödel's theorems. ## Full text (the author's own words) …I have no formal education in programming or in training LLMs. I got into building apps and training AI through a thoroughly strange chain of coincidences which, looking back, seems to make quite a lot of sense. And I think it says something important about our times. I hadn’t planned it; I never imagined I’d have more to do with technology than the average iPhone user. For years my only window onto the world of Big Tech was the Hard Fork podcast. Shout-out to Casey and Kevin :) It all started with vibe coding and the OrsonAI app. Playing around with AI, I began building tools that could help me in my day-to-day work as a journalist. Transcribing interviews while they were being recorded and importing them into an editor, or an editing mode that let me consult a chatbot about selected passages without switching windows and pasting in the whole context — at the time these were genuinely neat tools, and they sped my work up considerably. Over time I began to think of it as a project, something that could serve other people too. The idea was simple: take the features that save me time and build a product for people facing the same problems. I spent a lot of time building an interface and mechanisms that could deliver safe AI for newsrooms — AI that helps the journalist but doesn’t do the job for them. I even designed a system that issues certificates of human authorship for a text, published on the Hedera Hashgraph public ledger. An immutable record with evidentiary value in court, showing how the text came into being and what part a human played in writing it. - [Here you will find a description of the HDK technology](https://jakubcwirlej.com/articles/hdk-tech.md) - [Here you can download the free HDK library](https://jakubcwirlej.com/files/hdk/hdk-audit-0.1.3.zip) - [HDK on Hugging Face](https://huggingface.co/JakCwi/hdk-audit) But I was in for a big disappointment. The reception from the industry people I showed the app to was frosty, to say the least. The question that came up at the end of every presentation made me painfully aware of how wrong I’d been: “Why would anyone pay for your app when we’ve got ChatGPT for free?” Things like a certificate of human authorship, data protection or even a substantial improvement in workflow over what an ordinary chatbot offered didn’t matter. I heard this from people who, day in, day out, stress the importance of reliable journalism. Still, working on the app, and on an agentic AI system reliable enough to be used in newsrooms, pushed me in a completely new direction. Trying to understand better where language-model hallucinations come from, I started reading research papers on arXiv, and eventually designed and ran a series of studies on SwiGLU gate activations in several open language models. I wrote up the findings in a few papers that I posted on ResearchGate. Looking back, some of the claims in those papers strike me as megalomaniacal, and the writing itself would probably need revising and a more scientific vocabulary, but I’m not taking them down — if only for the sake of research integrity. Today, after many failed experiments and disappointments, and with more humility earned on the proving ground of language-model training, I would describe the GSI signals I found as knowledge proxy signals rather than a model’s “epistemic awareness”. On the other hand, hard data let me observe that confabulation is largely a two-stage process in which the model — having no grounding in facts — almost always chooses coherence of form over truth. For me personally, that was the most interesting moment of these experiments: the moment my old interest in philosophy and epistemology found an unexpected, fully measurable reflection in the architecture of transformers. How useful these signals are day to day, in production, is a separate question. Comparing how well they detect hallucinations against other methods in use would certainly require further research. The work consumed me completely. Perhaps that’s somewhat in keeping with my character: I like to throw myself into things that are a challenge but also let me learn something about the world around me and understand it better. Besides, before I went into journalism I studied philosophy, with a particular interest in the philosophy of science and epistemology. - [Here you will find an essay summing up some of my thoughts](https://www.researchgate.net/publication/397730862) Training my own model ended in total disaster. I probably made every mistake in the book, despite my habit of thorough research and the vast amount of specialist literature I’d devoured. The first results looked promising, but only later did I understand that they were largely down to noise regularisation. The model’s advantage in ablation tests didn’t justify the extra compute needed to calculate the Gini of the gate activations. On top of that came purely technical training challenges. When my modifications were exempted from normalisation, they held back effective learning, whereas normalising them flattened them out, turning the model into a standard transformer that was just more expensive to train. I’m making the architecture details available for download here — perhaps someone will find useful material in it for learning from other people’s mistakes. One solution that I think still deserves attention is the M2 gate. Thanks to an affine anchor, it can keep running on a sigmoid without the risk of being dragged down by a lazy optimiser looking for an easy way to lower the training loss. In my case this gate, trained on the Gini signals of the activations, was meant to cut noise during the model’s generation. However, taking the signal out from under normalisation — as I mentioned earlier — led to a model with an excessively peaked confidence distribution: logically sharper, but in practice worse at learning facts. Perhaps this solution could be useful in other models, for gating other components, and help the training process. In the end I came to see all my failures in training a model on my own architecture as capital that shouldn’t be wasted. Today I’m training my own small model, EUGEN 160M, which specialises in code. Comparing its benchmark results so far, I’m cautiously optimistic and expect this won’t be time wasted. I decided to train a model specialising in backend languages because with this kind of architecture it’s easier to monitor progress, and the model itself can be a useful autocomplete tool for inference on an ordinary CPU. You can try the model in a specially built terminal on my site and compare it with other open models of a similar size. - [Here you can try the Eugen 160M model](https://jakubcwirlej.com/ai-dev/#terminal) The results so far are above all down to rigorously tracking the model’s progress and the mistakes it makes during generation. That let me intervene on the fly, adding training data to strengthen the model wherever it showed weaknesses. I also regularly used the STaR and LEMA techniques, creating synthetic data from what the model itself had generated, as well as translation data that teaches the model the logic of code by showing it examples of translations between Python and C++, Java and Rust. All this material currently makes up about 0.35% of the whole dataset, and its share will be raised gradually as training goes on. Training the model is great fun and — like journalism before it — gives me the sense that I’m doing something important: something that lets me better grasp the changes in the world around me. I can’t write complex C++ from scratch on my own, that’s a fact, but I try to understand the logic of the processes. That’s how I chalk up the small wins that let me keep moving forward in my work. And yes — with the help of an AI coding agent I built my own Triton kernel that makes it possible to train a sparse model, improving TPS (Tokens Per Second) by 23% over a dense model; later kernel optimisations added a further 36%. I intend to use it to train my next, somewhat larger model. - [You can download the kernel here](https://jakubcwirlej.com/files/eugen-sparse.zip) - [Kernel on Hugging Face](https://huggingface.co/JakCwi/eugen-sparse) And coming back to the model where it all began... Today I can see that my biggest mistake wasn’t engineering naivety. That model simply had no chance of working the way I’d assumed. The problem wasn’t technological but purely philosophical. And as someone who studied philosophy, I should have caught it at the very start, with Tarski’s definition of truth and Gödel’s theorems in mind. According to them, no sufficiently complex system can fully prove its own consistency or define truth purely from within itself. Trying to force a model to analyse, evaluate and correct its own internal states in real time locked it into a structural loop from which — without a meta-level and an external point of reference — there was simply no way out. When I started out on this adventure, all I wanted was to build a tool that made transcribing journalistic interviews easier. But it very quickly turned out that entering the world of AI is a genuine plunge down the rabbit hole from “Alice in Wonderland”: the deeper you go, the more complex, layered and fascinating everything becomes. My first app may not have revolutionised the media market, but that fall down the rabbit hole has brought me to a place where I’m successfully training my own architecture. I’ve come a long way from passive tech-podcast listener to someone who physically shapes the parameters of models and understands what’s going on inside them. --- Jakub Ćwirlej — Journalist who builds with AI · on the site: https://jakubcwirlej.com/ai-dev/ Full context: https://jakubcwirlej.com/context/en.md · https://jakubcwirlej.com/llms.txt