Probability and statistics

10.1 Probability as measure

1. Theorem · A Additivity for overlapping sets If $f$ is finitely additive on a Boolean algebra, then $f(A \cup B) = f(A) + f(B \setminus A)$ and $f(A \cup B) = f(A) + f(B) - f(A \cap B)$ for all $A, B$ in the algebra.
2. Theorem · A Properties of a finitely additive measure For a finitely additive measure $f$: $f(A \cup B) \le f(A) + f(B)$; if $A \subseteq B$ then $f(B \setminus A) = f(B) - f(A)$ and $f(A) \le f(B)$; and $f(\varnothing) = 0$.
3. Theorem · B Vitali: no uniform probability on every subset There is no way to assign a probability to every subset of the interval [0, 1) so that probabilities add over countably many disjoint sets, the whole interval has probability 1, and sliding a set around the circle (mod 1) never changes its probability.
4. Theorem · A Basic rules of probability From the three axioms: $P(\varnothing) = 0$; probability is finitely additive; $P(A^c) = 1 - P(A)$; $A \subseteq B$ implies $P(A) \le P(B)$; every $P(A)$ lies in $[0, 1]$; and $P(A \cup B) = P(A) + P(B) - P(A \cap B)$.
5. Theorem · A Union bound For any events $A_1, A_2, \dots$, $P(A_1 \cup A_2 \cup \cdots) \le P(A_1) + P(A_2) + \cdots$ (overlaps are counted twice, so the sum can only be too big).
6. Theorem · A Continuity of probability If events grow, $A_1 \subseteq A_2 \subseteq \cdots$, then $P(A_n) \to P(\bigcup A_n)$. If events shrink, $A_1 \supseteq A_2 \supseteq \cdots$, then $P(A_n) \to P(\bigcap A_n)$.
7. Theorem · A Conditional probability is a probability measure
8. Theorem · A Law of total probability
9. Theorem · A Bayes' theorem
10. Theorem · A Complements of independent events If $A$ and $B$ are independent, so are $A$ and $B^c$, $A^c$ and $B$, and $A^c$ and $B^c$.
11. Theorem · A Properties of a distribution function $F$ is non-decreasing, tends to 0 at minus infinity and to 1 at plus infinity, is continuous from the right, satisfies $P(a < X \le b) = F(b) - F(a)$, and has $P(X = a) = F(a) - F(a^-)$, the size of its jump at $a$.
12. Theorem · A The exponential distribution forgets
13. Theorem · B The Gaussian integral $\displaystyle\int_{-\infty}^{\infty} e^{-x^2/2}\,dx = \sqrt{2\pi}$, so the standard normal density has total area 1.
14. Theorem · A Linearity of expectation For random variables on a countable sample space with finite expectation, $E[X + Y] = E[X] + E[Y]$ and $E[aX + b] = aE[X] + b$.
15. Theorem · A Variance shortcut and scaling $\operatorname{Var}(X) = E[X^2] - (E[X])^2$ and $\operatorname{Var}(aX + b) = a^2\operatorname{Var}(X)$.
16. Theorem · A Variance of a sum of independent variables If $X$ and $Y$ are independent discrete random variables with finite variance, then $E[XY] = E[X]E[Y]$ and $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$.
17. Theorem · A Markov's inequality
18. Theorem · A Chebyshev's inequality
19. Theorem · A Weak law of large numbers