This independent technical project documents the full process of training a German GPT-2-style language model: preparing a large German Common Crawl corpus, configuring the model and tokenizer, running distributed training, and evaluating the result.
The resulting model has 358 million parameters and a context window of 2,048 tokens. The project is primarily an engineering and learning experiment, rather than part of my main research program on reliable machine learning.
