Training a German LLM from Scratch
This article is not finished and will be updated. The research group I work with has access to a small GPU cluster, which occasionally sits idle. To avoid wasting valuable compute resources …
This independent technical project documents the full process of training a German GPT-2-style language model: preparing a large German Common Crawl corpus, configuring the model and tokenizer, running distributed training, and evaluating the result.
The resulting model has 358 million parameters and a context window of 2,048 tokens. The project is primarily an engineering and learning experiment, rather than part of my main research program on reliable machine learning.
