> For the complete documentation index, see [llms.txt](https://www.mlcompendium.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.mlcompendium.com/deep-learning/bayesian-neural-nets.md). # Bayesian Neural Nets A softmax score is not a confidence measure, so a network that forecasts or flags anomalies also needs to say how unsure it is. This page starts from what a BNN is and the three types of prediction uncertainty, then shows the practical trick of running prediction with dropout many times, how Uber applies it to time series, and how to do it in Keras. BNN - (what is?) [Bayesian neural network (BNN)](http://edwardlib.org/tutorials/bayesian-neural-network) is the starting reference. According to Uber, it is an architecture that more accurately forecasts time series predictions and uncertainty estimations at scale. “how Uber has successfully applied this model to large-scale time series anomaly detection, enabling better accommodate rider demand during high-traffic intervals.” The Uber post itself is kept at the end of the page. Under the BNN framework, prediction uncertainty can be categorized into three types: * Model uncertainty captures our ignorance of the model parameters and can be reduced as more samples are collected. * model misspecification * inherent noise captures the uncertainty in the data generation process and is irreducible. Note: in a series of articles, uber explains about time series and leads to a BNN architecture. The first step is Neural networks - training on multi-signal raw data, training X and Y are window-based and the window size(lag) is determined in advance. Vanilla LSTM did not work properly, therefore an architecture of (the note stops here; the encoder-decoder answer comes further down). Those Uber articles are kept at the end of the page. Regarding point 1: ‘run prediction with dropout 100 times’. The [MEDIUM with code how to do it.](https://medium.com/hal24k-techblog/how-to-generate-neural-network-confidence-intervals-with-keras-e4c0b78ebbdf) is Sam Blake's post on how to generate neural network confidence intervals with Keras. The reason to bother is the softmax itself. [Why do we need a confidence measure when we have a softmax probability layer?](https://hjweide.github.io/quantifying-uncertainty-in-neural-networks) The blog post explains, for example, that with a CNN of apples, oranges, cat and dogs, a non related example such as a frog image may influence the network to decide its an apple, therefore we can’t rely on the probability as a confidence measure. The ‘run prediction with dropout 100 times’ should give us a confidence measure because it draws each weight from a bernoulli distribution. “By applying dropout to all the weight layers in a neural network, we are essentially drawing each weight from a [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution). In practice, this mean that we can sample from the distribution by running several forward passes through the network. This is referred to as [Monte Carlo dropout](http://arxiv.org/abs/1506.02158).” The Monte Carlo dropout link is the paper Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference. Taken from Yarin Gal’s blog post (kept at the end of the page). In this figure we see how sporadic is the signal from a forward pass (black line) compared to a much cleaner signal from 100 dropout passes.

Is it applicable for time series? In the figure below he tried to predict the missing signal between each two dotted lines, A is a bad estimation, but with a dropout layer we can see that in most cases the signal is better predicted.

Going back to uber, they are actually using this idea to predict time series with LSTM, using encoder decoder framework.

Note: this is probably applicable in other types of networks. The theory behind it is the Phd Thesis by Yarin, where he talks about uncertainty in Neural networks and using BNNs. he may have proved this thesis, but I did not read it. This blog post links to his full Phd; both are kept at the end of the page. Old note: The idea behind uncertainty is ([paper here](https://arxiv.org/pdf/1506.02142.pdf)) that in order to trust your network’s classification, you drop some of the neurons during prediction, you do this \~100 times and you average the results. Intuitively this will give you confidence in your classification and increase your classification accuracy, because only a partial part of your network participated in the classification, randomly, 100 times. Please note that Softmax doesn't give you certainty. A Medium post on prediction with drop out was noted here as well. In Keras, the [solution for keras](https://github.com/keras-team/keras/issues/9412) says to add trainable=true for every dropout layer and add another drop out at the end of the model. Thanks sam. The snippet from that thread: “import keras inputs = keras.Input(shape=(10,)) x = keras.layers.Dense(3)(inputs) outputs = keras.layers.Dropout(0.5)(x, training=True) model = keras.Model(inputs, outputs)“ ## Deprecated links {% hint style="warning" %} These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above. {% endhint %} \- Towards Data Science: is-your-algorithm-confident-enough-1b20dfe2db08. This address no longer opens: - BNN. This address no longer opens: - Neural networks. This address no longer opens: - blog post. This address no longer opens: - Phd Thesis by Yarin. This address no longer opens: --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://www.mlcompendium.com/deep-learning/bayesian-neural-nets.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.