A Reality Check

Before reading any NLP leaderboard, it is worth asking whether the gains survive a fair comparison. The metric-learning reality check says many do not: once datasets are normalized, many algorithms show little metric progress. The page gives the benchmark, the paper, and the follow-up note that make that stall visible.

The same notes are in A metric learning reality check.

The tool behind the comparison is A powerful benchmark, a library for ML benchmarking. The argument itself is the paper, "A Metric Learning Reality Check", and the medium post carries the updates to it — together they show that normalizing data sets allows us to see that there wasn't any advancement in terms of metrics in many NLP algorithms.

Deprecated links

These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.

• A powerful benchmark This address no longer opens: https://github.com/KevinMusgrave/powerful-benchmarke