Named Entity Recognition is a solved problem — if you have tens of thousands of labeled examples. But what happens when you need to extract entities from a language that has maybe a few hundred annotated sentences, if any? This was the challenge the BSNLP 2021 shared task threw at us: NER across four Slavic languages under low- and zero-resource conditions.
The problem: parallel annotation doesn't exist
Unlike high-resource language pairs, Slavic languages don't have large parallel annotated corpora. You can't directly project labels from Polish to Ukrainian and expect good results. The morphology differs, entity boundaries shift, and transliteration inconsistencies break naive alignment.
Meta-learning to the rescue
We approached this with MAML-inspired meta-learning: train the model to adapt quickly to new entity types given just a handful of examples. The core idea is to optimize for parameters that require only a few gradient steps to specialize for a new language or domain.
The adaptation loop
- Sample a support set of 8-32 sentences from the target language
- Perform 3-5 inner-loop gradient updates on the support set
- Evaluate on a query set and backprop through the adaptation
- Update meta-parameters to minimize query-set loss across many episodes
Cross-lingual transfer via adversarial training
To bridge the gap between languages, we added an adversarial language discriminator to the shared encoder. During training, the encoder tries to fool the discriminator while still performing well on NER. This encourages language-invariant representations in the shared space.
The surprising finding: zero-shot transfer from Polish to Belarusian was better than from Russian to Belarusian, despite Russian being linguistically closer. The orthographic similarity of Polish and Belarusian apparently matters more than treebank-based linguistic distance metrics suggest.
Results and lessons
Our final submission placed second on the zero-shot track and first on the few-shot track. The main lesson: when data is scarce, architectural creativity matters more than compute scale. A 110M parameter model with careful meta-learning can outperform a 500M parameter model fine-tuned naively.
If you're working on low-resource NER, I'd recommend starting with cross-lingual word embeddings and gradually adding complexity only where simple baselines fail. It's easy to over-engineer solutions in this space.