---
title: "Training data"
author: "Jakub Ćwirlej"
id: training-data
language: en
url: "https://jakubcwirlej.com/articles/training-data.md"
page: "https://jakubcwirlej.com/ai-dev/"
translations:
  pl: "https://jakubcwirlej.com/pl/articles/training-data.md"
  fr: "https://jakubcwirlej.com/fr/articles/training-data.md"
  nl: "https://jakubcwirlej.com/nl/articles/training-data.md"
updated: 2026-10-14
words: 157
summary: "The EUGEN 160M training mix by Hugging Face repository: 14 rows covering 93.4% of the current token stream. Dolma 3 is the largest part (35.29%, including Stack-Edu and Common Crawl by topic), followed by The Stack v3 (15.44%) and commit histories from CommitPack and related datasets (10.46%). Own data – STaR solutions generated by EUGEN and verified by tests, plus code translation pairs – make up about 0.35% of the mix."
summary_type: editorial
---

# Training data

Jakub Ćwirlej · text language: EN · 157 words

## Summary (editorial)

The EUGEN 160M training mix by Hugging Face repository: 14 rows covering 93.4% of the current token stream. Dolma 3 is the largest part (35.29%, including Stack-Edu and Common Crawl by topic), followed by The Stack v3 (15.44%) and commit histories from CommitPack and related datasets (10.46%). Own data – STaR solutions generated by EUGEN and verified by tests, plus code translation pairs – make up about 0.35% of the mix.

## Key points (editorial)

- Every row gives the exact Hugging Face repository ids; sources without a confirmed HF origin are left out.
- Code from GitHub Code Clean is limited to permissive licences (MIT, Apache-2.0, BSD, ISC, Unlicense, CC0, Artistic-2.0).
- Maths comes from OpenMathReasoning, OpenR1-Math-220k and FineMath, among others.
- The corpus holds about 904.5 billion tokens, intended repetitions included.

## Full text (the author's own words)

| Dataset (Hugging Face) | Share | Licence / notes |
| --- | --- | --- |
| Dolma 3 (incl. Stack-Edu and Common Crawl by topic): allenai/dolma3_mix-6T-1025-7B, allenai/dolma3_pool, allenai/dolma3_dolmino_mix-100B-1125 | 35.29% | ODC-By |
| The Stack v3: HuggingFaceCode/stack-v3-train | 15.44% | ODC-By (compilation), honouring the BigCode opt-out |
| bigcode/commitpack, bigcode/commitpackft, project-themis/git-commits-merged, CarperAI/github-diffs-deduped | 10.46% | repositories under permissive licences; CommitPackFT and github-diffs-deduped MIT, git-commits-merged Apache-2.0 |
| codeparrot/github-code-clean | 6.63% | permissive licences only: MIT, Apache-2.0, BSD-2/3-Clause, ISC, Unlicense, CC0-1.0, Artistic-2.0 |
| common-pile/arxiv_papers, common-pile/stackexchange_filtered, common-pile/data_provenance_initiative_filtered | 5.64% | open licences (Common Pile) |
| nvidia/OpenMathReasoning, open-r1/OpenR1-Math-220k | 4.18% | CC BY 4.0 (OpenMathReasoning) |
| HuggingFaceTB/finemath (3+ and 4+ subsets) | 3.95% | ODC-By |
| allenai/peS2o | 3.32% | ODC-By |
| instructions and reasoning: nvidia/OpenMathInstruct-2, TinyGSM/TinyGSM, OpenCoder-LLM/opc-annealing-corpus, OpenCoder-LLM/opc-sft-stage2, AI-MO/NuminaMath-CoT, meta-math/MetaMathQA, microsoft/orca-math-word-problems-200k, ise-uiuc/Magicoder-OSS-Instruct-75K, nvidia/OpenCodeReasoning-2, PrimeIntellect/SYNTHETIC-1, open-r1/codeforces-cots, hkust-nlp/CodeIO-PyEdu-Reasoning, allenai/ai2_arc, allenai/qasc | 2.26% | per each dataset’s licence |
| books: allenai/dolma3_mix-6T-1025-7B, common-pile/project_gutenberg_filtered, common-pile/pre_1929_books_filtered, common-pile/library_of_congress | 1.84% | public domain / ODC-By |
| HuggingFaceFW/finewiki, hltcoe/megawika | 1.60% | per each dataset’s licence |
| open-thoughts/AgentTrove | 1.47% | successful trajectories only |
| mandarjoshi/trivia_qa, microsoft/ms_marco, hotpotqa/hotpot_qa, rungalileo/ragbench | 1.00% | 30% are own “no answer in context” examples built from these datasets |
| own data: STaR (EUGEN’s solutions verified by tests), learning from mistakes, translation pairs Python ↔ C++, Java, Go, Rust | 0.35% | own |

---

Jakub Ćwirlej — Journalist who builds with AI · on the site: https://jakubcwirlej.com/ai-dev/
Full context: https://jakubcwirlej.com/context/en.md · https://jakubcwirlej.com/llms.txt