Bayes Rule: Visual Referesher

Bayes rule is a familiar or natural outcome for most familiar with probability theory. In words, it tells us how to update the probability of a random variable(s) given some event(s) has occurred and that we have some prior knowledge or belief about the probability of the random variable(s) from earlier events. The algebra to get to Bayes rule is simple but I found it always best to have a more spatial perspective on what Bayes rule is really stating.

I'll first begin with ta 2D square sample space, $\it{S}$. This space is discrete and we can represent each outcome as a tiny square, $\it{s}$. In this case, we will have a total of 12 tiny squares in $\it{S}$. This means there is a 1/16 chance that any square is randomly selected, hence, $p(s)$.

$$\begin{array}{|c|c|c|c|}\hline  \it{s_1} & \it{s_2}  & \it{s_3}  & \it{s_4}   \\ \hline \it{s_5}  & \it{s_6}  & \it{s_7} & \it{s_8}  \\ \hline \it{s_9}  & \it{s_{10}}  & \it{s_{11}}  &  \it{s_{11}} \\ \hline \it{s_{13}}  & \it{s_{14}} & \it{s_{15}}  & \it{s_{16}} \\ \hline \end{array}$$

$$\mathrm{P}(\it{s}_i)_{\it{S}} = \mathrm{1/16}$$

Now we say have the scenario where we are only interested in two subspaces of $\it{S}$, $\it{S}_A$ and $\it{S}_B$. More specifically we want to know the probabilities of a square randomly occurring in each of these subspaces given they occur in $\it{S}$ and what the probability is of a square occurring in the intersection, or state differently, the probability of a square occurring in both $\it{S}_A$ and $\it{S}_B$.

With this we have the following: $\mathrm{P}(\it{s}_A)$, $\mathrm{P}(\it{s}_B)$, and $\mathrm{P}(\it{s}_A \cap \it{s}_B) = \mathrm{P}(\it{s}_A)$. The updated image of this would look like:

The probability $\mathrm{P}(\it{s}_A)$ in red, $\mathrm{P}(\it{s}_B)$ in blue, and the overlap $P(\it{s}_A \cap \it{s}_B)$. Keep in mind that $\mathrm{P}(\it{s}_A \cap \it{s}_B) = \mathrm{P}(\it{s}_A \cap \it{s}_B) = \mathrm{1/8}$. 

The question we usually want to ask is not what the joint probability, i.e., what's the probability of both $\it{S}_A$ and $\it{S}_B$ squares, but instead is what is the probability of a square in $\it{S}_A$  given that a square in $\it{S}_B$ has been picked/occurred or vice versa.  So what does this mean? We want to compare the relative probabilities of the joint space to that of the given space where the event has occurred:

\begin{equation} \mathrm{P}(\it{s}_A | \it{s}_B) = \frac{\mathrm{P}(\it{s}_A  \cap \it{s}_B)}{\mathrm{P}(\it{s}_B)}\label{eq:bayes1}   \end{equation} 

and

\begin{equation} \mathrm{P}(\it{s}_B| \it{s}_A) = \frac{\mathrm{P}(\it{s}_A  \cap \it{s}_B)}{\mathrm{P}(\it{s}_A)} \label{eq:bayes2}\end{equation}

Notice how these two equations are not the same but we the probability in the joint space, $\mathrm{P}(\it{S}_A \cap \it{S}_B) = \mathrm{P}(\it{S}_B \cap \it{S}_A)$. This had to be the case just by looking at the illustration with the colored cells above. 

The key is that we can now determine the conditional probabilities, that is the probability of a cell in a subspace given a cell in the other subspace has been picked or occurred, by rearrange eq. \ref{eq:bayes1} and eq. \ref{eq:bayes2} for the joint probability and then substituting terms to get:

\begin{equation*} \mathrm{P}(\it{s}_A | \it{s}_B) \mathrm{P}(\it{s}_B) = \mathrm{P}(\it{s}_B | \it{s}_A) \mathrm{P}(\it{s}_A)\end{equation*}

which is rearranged to get the typical Bayes formula:

\begin{equation}\mathrm{P}\left(\it{s}_A | \it{s}_B\right)  = \frac{\mathrm{P}\left(\it{s}_B | \it{s}_A\right) \mathrm{P}\left(\it{s}_A\right)}{\mathrm{P}\left(\it{s}_B\right)} \label{eq:bayesformula}\end{equation}.

At first eq. \ref{eq:bayesformula} might seem expected you could. I mean it is just an outcome of analyzing probabilities of subspaces, but the impact is really how one can this equation to update knowledge. Let us break down the terms in eq. \ref{eq:bayesformula}.

The first term in the numerator is called the likelihood probability. It indicates how probably an event in $\it{S}_B$ is given that an event in $\it{S}_A$ occurs. It can also represent the probability of the observed data given the model and its parameters (i.e. prior over parameters). The second term in the numerator, the prior, informs about previous knowledge of the observations or parameters. Finally, the denominator can be interpreted as the probability of observing a cell in $\it{S}_B$ or you can think about it as the data averaged over all possible values of the model parameters. 

An important aspect of eq. $\ref{eq:bayesformula}$ is that in the case of probability functions, the integration equals one. This just means that over the whole space of probabilities, something must have happened.

In the example given, the probabilities are just uniform discrete values, so we obtain a posterior probability that is just a number that represents our updated knowledge about the probability of a cell in $\it{S}_{A}$ given the cell is in $\it{S}_{B}$. This is a particularly simple and maybe intuitive outcome. What is typically more useful is that we have a probability density function that represents our prior knowledge about an event/outcome and we want to determine the posterior distribution. We then choose a likelihood probability that encodes information about what has been observed given the prior probability and make inferences by sampling the constructed posterior distribution.

Reuse and Attribution