Information Theory
A model that ranks classes or splits trees needs a measure of surprise and a measure of how wrong a predicted distribution is. The page starts with entropy and information gain, moves through the cross-entropy and divergence family and softmax, and then turns to tools, tutorials, time-series entropy, and complement objective training. The same notes are in Collocation and MUTUAL INFORMATION COEFFICIENT.
Entropy / Information Gain
Surprise comes first, because every later measure on the page is built from it.
The same notes are in Active Learning.
Shannon entropy in python is, basically, entropy(value counts): count how often each value appears, turn the counts into probabilities, and sum p log p. MachineLearningMastery's "A Gentle Introduction to Information Entropy" is the Mastery on plogp entropy function walkthrough, and the Entropy functions gist shows four different ways to calculate entropy in Python.
Cross entropy, relative ent, KL-D, JS-D, soft max
Entropy measures one distribution; training needs a measure between two, the true one and the predicted one.
The same notes are in Comparing distributions (distance methods), LOSS, and LOSS IN KERAS.
Kullback–Leibler divergence is a very useful way to measure the difference between two probability distributions, and Count Bayesie's "Kullback-Leibler Divergence Explained" is A really good explanation on all of them. From the loss side, Raúl Gómez Bruballa's post on categorical and binary cross-entropy, softmax loss, logistic loss, focal loss, and all those confusing names is Another good one on all of them. MachineLearningMastery's "A Gentle Introduction to Cross-Entropy for Machine Learning" is the mastery on a gentle intro to CE, and the Mastery on entropy follow-up covers divergence between distributions: kullback leibler divergence (asymmetry), jensen-shannon divergence (symmetry) (has code).
The same family is tied together in Entropy, mutual information and KL Divergence by AurelienGeron. Gensim also has notes on divergence metrics such as KL jaccard etc, pros and cons, lda is a mess on small data. For computing it by hand, the Stack Exchange question on calculating KL divergence in Python, where every pair of lists returned the same value, is the Advise on KLDivergence thread. Neural machine translation using pytorch and CE is where the same loss shows up in a full model.
Softmax
Cross-entropy needs a predicted distribution to compare against, and softmax is what turns a network's scores into one.
The same notes are in ACTIVATION FUNCTIONS and Temperature.
Understanding softmax is Uniqtech Learning's short explanation of the activation function that turns logits into probabilities that sum to one. LJ V. MIRANDA's notebook, Softmax and negative likelihood (NLL), explains the softmax function, its relationship with the negative log-likelihood, and its derivative during backpropagation. The naming question is settled in Softmax vs cross entropy - Softmax loss and cross-entropy loss terms are used interchangeably in industry. Technically, there is no term as such Softmax loss. people use the term "softmax loss" when referring to "cross-entropy loss". The softmax classifier is a linear classifier that uses the cross-entropy loss function. In other words, the gradient of the above function tells a softmax classifier how exactly to update its weights using some optimization like gradient descent.
The softmax() part simply normalises your network predictions so that they can be interpreted as probabilities. Once your network is predicting a probability distribution over labels for each input, the log loss is equivalent to the cross entropy between the true label distribution and the network predictions. As the name suggests, softmax function is a “soft” version of max function. Instead of selecting one maximum value, it breaks the whole (1) with maximal element getting the largest portion of the distribution, but other smaller elements getting some of it as well.
This property of softmax function that it outputs a probability distribution makes it suitable for probabilistic interpretation in classification tasks.
Cross entropy indicates the distance between what the model believes the output distribution should be, and what the original distribution is. Cross entropy measure is a widely used alternative of squared error. It is used when node activations can be understood as representing the probability that each hypothesis might be true, i.e. when the output is a probability distribution. Thus it is used as a loss function in neural networks which have softmax activations in the output layer.
Tools
Once the measures are clear, computing them should not mean writing them from scratch. EntroPy is the entropy 0.1.3 documentation, and AntroPy has its own 0.2.2 documentation; the AntroPy [Git repo describes it as entropy and complexity of (EEG) time-series in Python. PyInform [Docs]- PyInform is a python library of information-theoretic measures for time series data. PyInform is backed by the Inform C library, a cross platform library for information analysis of dynamical systems. PyEntropy is one more package, entropy for Python.
Tutorials
The libraries compute the numbers; the tutorials show why a decision tree wants them.
The same notes are in CART TREES.
The scikit-learn decision tree learning tutorial on entropy, Gini, and information gain is the Great tutorial on all of these topics.
Entropy - lack of order or lack of predictability (excellent slide lecture by Aurelien Geron). The figures below come from that short introduction to entropy, cross-entropy, and KL-divergence.
Cross entropy will be equal to entropy if the probability distributions of p (true) and q(predicted) are the same. However, if cross entropy is bigger (known as relative_entropy or kullback leibler divergence)
In this example we want the cross entropy loss to be zero, i.e., when we have a one hot vector and a predicted vector which are identical, i.e., 100% in the same class for predicted and true, we get 0. In all other cases we get some number that gets larger if the predicted class probability is lower than zero as seen here:
Formula for 2 classes:
NOTE: Entropy can be generalized as a formula for N > 2 classes:
The same loss logic now carries over to trees. The notes below follow an awesome pdf tutorial whose address no longer opens and is kept at the end of the page. (We want to grow a simple tree): a good attribute prefers attributes that split the data so that each successor node is as pure as possible, i.e., the distribution of examples in each node is so that it mostly contains examples of a single class. In other words: We want a measure that prefers attributes that have a high degree of „order“. Maximum order: All examples are of the same class. Minimum order: All classes are equally likely → Entropy is a measure for (un-)orderedness. Another interpretation: Entropy is the amount of information that is contained, and all examples of the same class → no information.
Entropy is the amount of unorderedness in the class distribution of S. In the IMAGE above it has its maximal value when the equal class distribution holds, and its minimal value when only one class is in S.
That is the entropy of one node; a split has one node per category. So basically if we have the outlook attribute and it has 3 categories, we calculate the entropy for E(feature=category) for all 3.
INFORMATION: The I(S,A) formula below.
What we actually want is the average entropy of the entire split, that corresponds to an entire attribute, i.e., OUTLOOK (sunny & overcast & rainy)
Information Gain: is actually what we gain by subtracting information from the entropy.
In other words we find the attributes that maximizes that difference, in other other words, the attribute that reduces the unorderness / lack of order / lack of predictability.
The BIGGER GAIN is selected.
There are some properties to Entropy that influence INFO GAIN (?):
There are some disadvantages with INFO GAIN, done use it when an attribute has many number values, such as “day” (date wise) 05/07, 06/07, 07/07..31/07 etc.
Information gain is biased towards choosing attributes with a large number of values, and that bias causes overfitting and fragmentation.
We measure Intrinsic information of an attribute, i.e., Attributes with higher intrinsic information are less useful.
We define Gain Ratio as info-gain with less bias toward multi value attributes, ie., “days”
NOTE: Day attribute would still win with the Gain Ratio, Nevertheless: Gain ratio is more reliable than Information Gain
Therefore, we define the alternative, which is the GINI INDEX. It measures impurity, we define the average Gini, and the Gini Gain.
FINALLY, further reading about decision trees and examples of INFOGAIN and GINI here used to point at the same pdf tutorial, which is kept at the end of the page.
Beyond trees, mutual information itself is hard to estimate, and Variational bounds on mutual information is the arXiv paper "On Variational Bounds of Mutual Information".
Time series entropy
The tree measures treat each row on its own; a time series asks how predictable the sequence is.
The same notes are in Timeseries.
entroPY - EntroPy is a Python 3 package providing several time-efficient algorithms for computing the complexity of one-dimensional time-series. It can be used for example to extract features from EEG signals. The Approximate entropy paper is behind one of its measures, approximate entropy. The calls below compute each of them on a series x:
For measures beyond those, the PyInform 0.2.0 documentation lists the information-theoretic measures shown in the two figures below.
Complement Objective Training
Cross-entropy, from earlier on the page, only pushes up the correct class; complement objective training also uses the wrong ones. Complement Objective Training is a simple way to use incorrect-class probabilities and get more from labeled data in PyTorch Lightning. Article by LightTag, and the method's paper is on arXiv.
COT is a technique to effectively provide explicit negative feedback to our model. The technique gives us non-zero gradients with respect to incorrect classes, which are used to update the model's parameters.
COT doesn't replace cross-entropy. It's used as a second training step as follows: We run cross-entropy, and then we do a COT step. We minimize the cross-entropy between our target distribution. That's equivalent to maximizing the likelihood of the correct class. During the COT step, we maximize the entropy of the complement distribution. We pretend that the correct class isn't an option and make the remaining classes equally likely.
But, since the true class is an option, and we're training for it explicitly, maximizing the true classes probability and pushing the remaining classes to be equally likely is actually pushing their probabilities to 0 explicitly, which provides explicit gradients to propagate through our model.
Deprecated links
These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.
Entropy. This address no longer opens: https://www.techleer.com/articles/496-a-short-introduction-to-entropy-cross-entropy-and-kl-divergence-aurelien-geron/
awesome pdf tutorial. This address no longer opens: http://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf
FINALLY, further reading about decision trees and examples of INFOGAIN and GINI here.. This address no longer opens: http://www.ke.tu-darmstadt.de/lehre/archiv/ws0809/mldm/dt.pdf
Last updated
Was this helpful?