I've co-organized PolEval, the Polish NLP evaluation campaign, for three consecutive years. Running a shared task sounds glamorous — you set a problem, people solve it, you rank them. The reality is a constant battle against data leakage, ambiguous guidelines, and submission formats that break at 11:59 PM on deadline day.
Designing the task
A good shared task needs a Goldilocks difficulty: not so easy that existing baselines solve it perfectly, not so hard that nobody makes progress. For PolEval 2021's punctuation restoration task, we chose a real-world problem (restoring punctuation from ASR output) with enough inherent ambiguity to make it interesting.
The data leakage problem
Data leakage is the silent killer of shared task integrity. Contestants don't need to be malicious — training on Wikipedia and then evaluating on a Wikipedia-derived test set creates invisible overlap. We now use aggressive deduplication, temporal filtering (train before 2018, test after 2019), and n-gram overlap detection before releasing any split.
- MinHash Jaccard similarity threshold of 0.5 for passage-level deduplication
- Temporal train/test splits for chronologically-ordered data
- Manual spot-checking of top-scoring submissions for suspicious patterns
- Release test inputs without labels, run evaluation on our servers
Contestant experience matters
The most technically perfect evaluation is useless if participants can't submit. We learned to provide starter code, Docker images with pinned dependencies, and a local evaluation script that exactly mirrors our server-side scoring. The friction of participating correlates inversely with the quality and quantity of submissions.
One year we required a custom JSON format for submissions. The resulting parsing errors consumed 20% of my organizer time. Never again. We now accept the simplest format that unambiguously captures the required information.
Final thoughts
If you're thinking about organizing a shared task, my advice is: start small, over-communicate, and automate everything. The human hours you invest in infrastructure pay dividends in fairness, reproducibility, and your own sanity. And publish your evaluation scripts alongside the results — reproducibility starts with the organizers.