Benchmarking
A score means little until it is compared with something: another dataset, another algorithm, another machine. The page walks those comparisons in order, from datasets and classifiers through NumPy's BLAS, hardware, cloud providers, and deep-learning platforms, then NLP and multi-task learning leaderboards, and ends with notes on scaling networks.
Algorithms
The comparisons start with what is being run, the data and the algorithm, before the machine it runs on.
Datasets
The first benchmark is of the data itself. EFF FF Benchmarks in AI used to be the pointer here; that address no longer opens and is kept at the end of the page. On the library side, scikit bench is described in its own words: "scikit-learn_bench benchmarks various implementations of machine learning algorithms across data analytics frameworks. It currently support the scikit-learn, DAAL4PY, cuML, and XGBoost frameworks for commonly used machine learning algorithms."
Algorithms (classifiers)
With the implementations benchmarked, the next question is which classifier to pick. Comparing is the side-by-side of accuracy, speed, memory and 2D visualization of classifiers, and every model in it links to its scikit-learn page, each with its own gallery of examples:
SVM, k-nearest neighbors, Random Forest, AdaBoost Classifier, Gradient Boosting, Naive, Bayes, LDA, QDA, RBMs, Logistic Regression, RBM + Logistic Regression Classifier
The SVC, KNeighborsClassifier, RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier, GaussianNB, and LogisticRegression pages all list the classifier comparison among their examples, LDA and QDA point at the scikit-learn 0.16.1 documentation, and BernoulliRBM shows Restricted Boltzmann Machine features for digit classification. The same comparison holds inside one model family: LSTM vs cuDNN LSTM shows that batch size of power 2 matters, and the latter is faster.
Numpy Blas
Under every classifier sits linear algebra, so the library NumPy links against changes the timing. How do i know which version of blas is installed is the question of how to be sure NumPy uses a BLAS library at all when it was installed from a package manager rather than compiled by hand. Benchmark OpenBLAS, Intel MKL vs ATLAS is a shared benchmark of LAPACK / BLAS libraries from someone replacing CPU and math library for a huge simulation model, who concluded that Intel MKL is the best. The figure is that benchmark.
Another comparison of NumPy BLAS builds is in the next figure; its source post no longer opens and is kept at the end of the page. Both figures are copied from the original hosted images.
Hardware
Past the libraries, the machine decides the rest, and the first choice is GPU versus CPU. Nvidia compares the 1070 vs 1080 vs 2080. Cpu vs GPU benchmarking for CNN/Test/LTSM/BDLTSM benchmarks TensorFlow on cloud CPUs and finds them cheaper for deep learning than cloud GPUs, because of the massive cost differential afforded by preemptible instances. Nvidia GPUs is TitanXp vs GTX1080Ti for Machine Learning: the GTX1080Ti proved as good as the Titan X Pascal at a much lower price, and the new Titan Xp is tested to see how much faster it is. As of March/17, Nvidia GPUs for desktop compares cards in terms of price and cuda units, and the bottom line is 1060-1080. Another bench up to 2013 is regarding many GPUS vs CPUs in terms of BW.
Cloud providers
Renting the hardware brings its own comparison, the hardware benchmarks across cloud providers from the gensim series. Part 1 and part 2 y gensim were that series; both addresses now open PII Tools, a product for analyzing personal data and sensitive information at scale, not the benchmarks.
Platforms
On a given machine, the framework is the last variable. Cntk vs tensorflow was one comparison; its address no longer opens and is kept at the end of the page. CNTK, TEnsor, torch, etc on cpu and gpu is the paper Benchmarking State-of-the-Art Deep Learning Software Tools.
NLP
Leaderboards move the comparison from speed to capability, and language has its own. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization is a benchmark for the cross-lingual generalization ability of pre-trained multilingual models that covers 40 typologically diverse languages and includes nine tasks.
Multi-Task Learning
A multi-task benchmark raises the question of how one model should balance the tasks, through loss weighting and shared representations. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics (Yarin Gal), with an unofficial implementation on GitHub, states the problem: "In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task’s loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. " Ruder on Multi Task Learning gives the reason to share at all: "By sharing representations between related tasks, we can enable our model to generalize better on our original task. This approach is called Multi-Task Learning (MTL) and will be the topic of this blog post."
GLUE
For English understanding, the standard leaderboard is GLUE and its successor. The same notes are in BERT.
Glue / super glue is the leaderboard of the General Language Understanding Evaluation (GLUE) benchmark, a collection of resources for training, evaluating, and analyzing natural language understanding systems.
State of the art in AI
Beyond single leaderboards, state of the art can be tracked across the field, in terms of domain X datasets.
Scaling networks and predicting performance of NN
The last comparison is forward-looking: predicting train time and accuracy before scaling networks across GPUs. A great overview of NN types is the video for it, but the idea behind the video is to create a system that can predict train time and possibly accuracy when scaling networks using multiple GPUs, there is also a nice slide about general hardware recommendations. The figure below is that slide, copied from the original hosted image.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
Another comparison (NumPy BLAS). This address no longer opens: http://markus-beuckelmann.de/blog/boosting-numpy-blas.html
EFF FF Benchmarks in AI. This address no longer opens: https://www.eff.org/ai/metrics
Cntk vs tensorflow. This address no longer opens: http://minimaxir.com/2017/06/keras-cntk/
Last updated
Was this helpful?