A probability distribution is an assignment of probabilities to events within a mathematical space.
The mathematical space is made up of atomic, mutually exclusive outcomes. These outcomes are the points in this space.
An event is a set of these outcomes. Two events may overlap and therefore they are not always mutually exclusive. A probability distribution assigns probabilities to these events.
Mathematically, probability distributions are often represented as triples:
The full mathematical space, represented by the capital Greek letter Omega
The set of events (which are subsets of Omega, since not all subsets may receive a probability)
The actual function P that assigns probability values to events
Properties of the Probability Distribution
Any random assignment of probabilities to events does not make a probability distribution. It must have some logical consistency.
The classic example is if one event is a subevent of another. The value of P(it rains tomorrow) must be greater than or equal to the value of P(there will be precipitation tomorrow), because the former contains all outcomes of the latter.
Another property is that the probability of the event of the entire mathematical space is 1, and the probability of the empty set is 0.
Mathematicians usually require that probability distributions follow the Kolmogorov Axioms, which are a few rules that require consistency in the probability assignments.
There are arguments for relaxing some of Kolmogorov’s Rules. One such rule is that the probabilities are required to be countably additive, which precludes distributions that cannot be normalized. Sometimes, improper distributions are useful to analyze, such as the fair countable lottery.
Finite Probability Distributions
The potentially complicated nature of probability distributions goes away when there is a finite number of outcomes to consider (such as a coin toss or a dice roll).
The categorical distribution is the simplest and most general of these finite distributions - which assigns a probability to every single outcome (which all add to 1). Every event is therefore just assigned the sum of the probabilities of its individual outcomes.
Continuous Probability Distributions
Many distributions are continuous, and common examples include the normal distribution, the continuous uniform distribution, and the beta distribution.
In these cases, probability can be assigned to subsets (and especially intervals) of real numbers. For example, in a normal distribution with mean 0, the probability of getting a value above the mean is 50% (the mean and median are equal in this case). The probability of being within 1-standard deviation of the mean is approximately 68%.
In most examples, the probability assigned to any given point is 0, but the point also have a value in a probability distribution function. This PDF is set up so that definite integrals over the PDF return the probability of that range. The PDF value at a point could also be thought of as the probability density at that point.