Probability & Statistics

Probability asks what happens next under a random process; statistics asks what process would explain what already happened. The page starts from that difference, then walks introductions to probability and statistics, more statistics, sampling and resampling, wiki topics, and recommended courses. The same notes are in Probability.

The best single starting point is a Coursera course on probabilities - for data science, actually quite good in explaining a lot of the basic tools,prob, conditional, distributions, sampling, CI, hypothesis, etc. A great resource for proba/bayes/b-networks/etc (adam bali) used to sit beside it; that address no longer opens and is kept at the end of the page.

Difference between

Before any tool, it helps to see that the two fields invert the same uncertainty problem. The Stack Exchange question Difference between asks what separates probability from statistics and why they are studied together. I.e, Probability deals with predicting the likelihood of future events, while statistics involves the analysis of the frequency of past events. The problems considered by probability and statistics are inverse to each other.

In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens.

=> Underlying process + randomness and random variables -> what happens next?

In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.

=> observe what happened -> what is the underlying process?

Finally, probability theory is mainly concerned with the deductive part, statistics with the inductive part of modeling processes with uncertainty.

Introduction to Probability

The deductive side comes first, and Math is Fun walks it one small page at a time, starting from the observation that life is full of random events. Types of events names them, Independent events covers events that do not affect each other, and Conditional proba is how to handle dependent events. Calculating probabilities can be hard, sometimes we add them, sometimes we multiply them, and often it is hard to figure out what to do, which is why Proba tree diagrams come next. Mutually exclusive events are the ones that cannot happen at the same time, like turning left and turning right, or heads and tails.

Counting comes before most of those calculations. In English we use the word combination loosely, without thinking if the order of things is important, and Combination and permutations makes that distinction exact. Bayes is the Math is Fun page on Bayes' theorem. Least squares regresssion It works by making the total of the square of the errors as small as possible (that is why it is called "least squares".

The sequence ends with random variables. A Random Variable is a set of possible values from a random experiment, such as heads and tails given the values 0 and 1. Random variables introduces them, Continuous random variables extends them past discrete outcomes, and Random vars mean, std, variance gives their mean, variance, and standard deviation.

Introduction to statistics

With random variables in hand, the inductive side starts from data itself. Data is a collection of facts and numbers, and the Math is Fun Table of contents for using and handling data is the index for everything below.

The first statistics are centers. The Median is the middle of a sorted list of numbers (Median), and the mode is the number which appears most often (Mode). The mean has several forms: the Weighted mean; the Geometric mean, a special type of average where we multiply the numbers together and then take a square root (for two numbers), cube root, and so on; and the Harmonic mean, the reciprocal of the average of the reciprocals.

Centers then need spread and position. Percentile is the value below which a percentage of data falls (Percentiles). Mean deviation is how far, on average, all values are from the middle. When two sets of data are strongly linked together we say they have a High Correlation (Correlation). Deviation means how far from the normal, and the Standard deviation is a measure of how spread out numbers are; the formula page gives the calculation, deviation meaning how far from the average.

Spread leads to shape. Data can be distributed (spread out) in different ways, and in many cases it tends to sit around a central value, which is the Standard normal distribution. Data can be skewed, meaning it tends to have a long tail on one side or the other, and Skewness of distribution explains why it is called negative skew.

Shape then supports inference. A confidence interval is a range of values we are fairly sure our true value lies in: Confidence intervals (using std). Accuracy is how close a measured value is to the true value and precision is how close the measurements are to each other, the Accuracy vs precision (accurate vs hitting closely or density) page. Probability is how likely something is to happen, and the complement of an event is all outcomes that are NOT the event (Probability complement). Is that a true difference, or just a fluke? The Chi-square test, p_value, independent, dependent, significance page answers that by telling a real relationship from random chance.

Two terms get confused often enough to need their own links. Variation vs variance - a private case - is the Stack Exchange question on whether variation and variance are the same term. Std vs variance - std is in the same metric as the mean, is the root of variance., allows outliers to influence, will not result in samples cancelling each other without the square root in the formula.

More on Statistics

After the introductions, the same concepts come back in plain English as a Data Science Central series, in three parts. The first is 25 concepts, then (part 2) 29 more concepts, then (part1) & part 3.

STATISTICAL SAMPLING AND RESAMPLING

Every statistic above is computed on a sample, so the next question is how the sample was drawn and how wrong it can be.

The same notes are in TRAIN / TEST / CROSS VALIDATION.

MachineLearningMastery's "A Gentle Introduction to Statistical Sampling and Resampling" answers What is? Method for sampling/resampling, and sampling errors explained.

Another great course on probability, distribution types, conditional, joint, chain, etc. http://legacydirs.umiacs.umd.edu/~jbg/teaching/INST_414/

Wiki

Sampling from more than one variable at once needs the joint vocabulary, and the Wikipedia entries give the definitions. Marginal probability is the marginal distribution entry, Joint probability is the joint probability distribution entry, and Conditional probability points at the main Probability entry. The chain rule has two meanings worth separating: the probability [Chain rule](https://en.wikipedia.org/wiki/Chain_rule_(probability) on Wikipedia, and derivatives using the chain rule, on khan.

Recommended Courses

The shelf closes with courses and one friendly article. Another great course on probability, distribution types, conditional, joint, chain, etc. is the course listed under sampling above. Kahn is the probability course on the same site as the khan chain-rule video. Brandon Rohrer's "How Bayes Theorem works" is A really good intro. What are confidence intervals? is a very friendly introduction that discusses only the general idea, without too much fancy statistics terms, and with python.

(another angle) The main difference between probability and statistics has to do with knowledge: what are the known facts? Inherent in both probability and statistics is a population, every individual we are interested in studying, and a sample, consisting of the individuals that are selected from the population. In probability, we would start with us knowing everything about the composition of a population, and then would ask, “What is the likelihood that a selection, or sample, from the population, has certain characteristics?” In statistics, we have no knowledge about the types of socks in the drawer; we infer properties about the population on the basis of a random sample.

Some calculations to get you into probability:

• Finding out the probability of an event

• Of two consecutive events (multiplication)

• Of several events (sum)

• Etc..

Deprecated links

These links and images no longer work. The original wording is kept here. A same-resource copy, when one was checked, is used above.

• A great resource for proba/bayes/b-networks/etc (adam bali). This address no longer opens: https://metacademy.org/graphs/concepts/bayesian_networks#focus=i9mo2e09&mode=learn

• What are confidence intervals? This address no longer opens: https://towardsdatascience.com/a-very-friendly-introduction-to-confidence-intervals-9add126e714