Pre-training a transformer from scratch is one of those tasks that sounds straightforward on paper and then eats three months of your life. This post is a field report from training a Polish ELECTRA model — what worked, what didn't, and the subtle traps that cost me weeks.
Why ELECTRA?
ELECTRA's replaced-token detection objective is significantly more sample-efficient than MLM. For a mid-resource language like Polish, where every GPU-hour counts, this matters. Instead of masking 15% of tokens and predicting them, ELECTRA trains a discriminator to detect which tokens were replaced by a small generator network. Every token in the sequence becomes a training signal.
Data curation is 80% of the work
I started with the Polish subset of Oscar + Wikipedia dumps, then spent far too long on deduplication and quality filtering. The key insight: a smaller, cleaner corpus beats a larger noisy one. My final training set was ~25GB of text, down from an initial ~120GB.
- Remove boilerplate with MinHash LSH deduplication
- Filter by perplexity score against a small n-gram model
- Strip markup and normalize quotes/dashes
- Segment into coherent passages, not random 512-token chunks
Generator-discriminator balance
The original ELECTRA paper uses a generator that's 1/3 the size of the discriminator. I found that for Polish, a 1/4 ratio worked better — possibly because the morphological richness of Polish makes the generation task harder, and a weaker generator actually produces more educational errors for the discriminator to learn from.
The biggest mistake was spending two weeks tuning the learning rate while the real problem was a bug in my data loader that silently dropped 30% of sentences.
Hardware reality check
Training on a single 8×V100 node (DGX-1) for about three weeks got me to a model that outperformed multilingual mBERT on Polish NER and sentiment tasks. Was it worth it? For the learning experience, absolutely. For production, I'd probably fine-tune a larger multilingual model today. But there's something deeply satisfying about training a model that truly understands Polish morphology from first principles.
The full model weights and training code are available on HuggingFace Hub. Feel free to reach out if you're attempting something similar — I'm happy to share the gritty details that didn't fit in this post.