Kullback–Leibler divergence
In mathematical statistics, the Kullback–Leibler (KL) divergence (also called relative entropy and I-divergence[1]), denoted , is a type of statistical distance: a measure of how one probability distribution P is different from a second, reference probability distribution Q.[2][3] Mathematically, it is defined as
A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model instead of P when the actual distribution is P. While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a metric, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions (in contrast to variation of information), and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence,[4] a generalization of squared distance, and for certain classes of distributions (notably an exponential family), it satisfies a generalized Pythagorean theorem (which applies to squared distances).[5]
Relative entropy is always a non-negative real number, with value 0 if and only if the two distributions in question are identical. It has diverse applications, both theoretical, such as characterizing the relative (Shannon) entropy in information systems, randomness in continuous time-series, and information gain when comparing statistical models of inference; and practical, such as applied statistics, fluid mechanics, neuroscience, bioinformatics, and machine learning.
Introduction and context
[edit]Consider two probability distributions P and Q. Usually, P represents the data, the observations, or a measured probability distribution. Distribution Q represents instead a theory, a model, a description or an approximation of P. The Kullback–Leibler divergence is then interpreted as the average difference of the number of bits required for encoding samples of P using a code optimized for Q rather than one optimized for P. Note that the roles of P and Q can be reversed in some situations where that is easier to compute, such as with the expectation–maximization algorithm (EM) and evidence lower bound (ELBO) computations.
Etymology
[edit]The relative entropy was introduced by Solomon Kullback and Richard Leibler in Kullback & Leibler (1951) as "the mean information for discrimination between and per observation from ",[6] where one is comparing two probability measures , and are the hypotheses that one is selecting from measure (respectively). They denoted this by , and defined the "'divergence' between and " as the symmetrized quantity , which had already been defined and used by Harold Jeffreys in 1948.[7] In Kullback (1959), the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions;[8] Kullback preferred the term discrimination information.[9] The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality.[10] Numerous references to earlier uses of the symmetrized divergence and to other statistical distances are given in Kullback (1959, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as the Jeffreys divergence.
Definition
[edit]For discrete probability distributions P and Q defined on the same sample space, the relative entropy from Q to P is defined[11] to be
which is equivalent to
In other words, it is the expectation of the logarithmic difference between the probabilities P and Q, where the expectation is taken using the probabilities P.
Relative entropy is only defined in this way if, for all x, implies (absolute continuity). Otherwise, it is often defined as ,[1] but the value is possible even if everywhere,[12][13] provided that is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.
Whenever is zero the contribution of the corresponding term is interpreted as zero because
For distributions P and Q of a continuous random variable, relative entropy is defined to be the integral[14]
where p and q denote the probability densities of P and Q.
More generally, if P and Q are probability measures on a measurable space and P is absolutely continuous with respect to Q, then the relative entropy from Q to P is defined as
where is the Radon–Nikodym derivative of P with respect to Q, i.e. the unique Q almost everywhere defined function r on such that which exists because P is absolutely continuous with respect to Q. Also we assume the expression on the right-hand side exists. Equivalently (by the chain rule), this can be written as
which is the entropy of P relative to Q. Continuing in this case, if is any measure on for which densities p and q with and exist (meaning that P and Q are both absolutely continuous with respect to ), then the relative entropy from Q to P is given as
Note that such a measure for which densities can be defined always exists, since one can take although in practice it will usually be one that in the context like counting measure for discrete distributions, or Lebesgue measure or a convenient variant thereof like Gaussian measure or the uniform measure on the sphere, Haar measure on a Lie group etc. for continuous distributions. The logarithms in these formulae are usually taken to base 2 if information is measured in units of bits, or to base e if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm.
Various conventions exist for referring to in words. Often it is referred to as the divergence between P and Q, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of P from Q or as the divergence from Q to P. This reflects the asymmetry in Bayesian inference, which starts from a prior Q and updates to the posterior P. Another common way to refer to is as the relative entropy of P with respect to Q or the information gain from P over Q.
Basic example
[edit]Kullback[3] gives the following example (Table 2.1, Example 2.1). Let P and Q be the distributions shown in the table and figure. P is the distribution on the left side of the figure, a binomial distribution with and . Q is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes 0, 1, 2 (i.e. ), each with probability .
![](http://upload.wikimedia.org/wikipedia/commons/thumb/8/8e/Kullback%E2%80%93Leibler_distributions_example_1.svg/400px-Kullback%E2%80%93Leibler_distributions_example_1.svg.png)
x | 0 | 1 | 2 |
---|---|---|---|
Distribution | |||
Distribution |
Relative entropies and are calculated as follows. This example uses the natural log with base e, designated ln to get results in nats (see units of information):
Interpretations
[edit]Statistics
[edit]In the field of statistics, the Neyman–Pearson lemma states that the most powerful way to distinguish between the two distributions P and Q based on an observation Y (drawn from one of them) is through the log of the ratio of their likelihoods: . The KL divergence is the expected value of this statistic if Y is actually drawn from P. Kullback motivated the statistic as an expected log likelihood ratio.[15]
Coding
[edit]In the context of coding theory, can be constructed by measuring the expected number of extra bits required to code samples from P using a code optimized for Q rather than the code optimized for P.
Inference
[edit]In the context of machine learning, is often called the information gain achieved if P would be used instead of Q which is currently used. By analogy with information theory, it is called the relative entropy of P with respect to Q.
Expressed in the language of Bayesian inference, is a measure of the information gained by revising one's beliefs from the prior probability distribution Q to the posterior probability distribution P. In other words, it is the amount of information lost when Q is used to approximate P.[16]
Information geometry
[edit]In applications, P typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while Q typically represents a theory, model, description, or approximation of P. In order to find a distribution Q that is closest to P, we can minimize the KL divergence and compute an information projection.
While it is a statistical distance, it is not a metric, the most familiar type of distance, but instead it is a divergence.[4] While metrics are symmetric and generalize linear distance, satisfying the triangle inequality, divergences are asymmetric and generalize squared distance, in some cases satisfying a generalized Pythagorean theorem. In general does not equal , and the asymmetry is an important part of the geometry.[4] The infinitesimal form of relative entropy, specifically its Hessian, gives a metric tensor that equals the Fisher information metric; see § Fisher information metric. Relative entropy satisfies a generalized Pythagorean theorem for exponential families (geometrically interpreted as dually flat manifolds), and this allows one to minimize relative entropy by geometric means, for example by information projection and in maximum likelihood estimation.[5]
The relative entropy is the Bregman divergence generated by the negative entropy, but it is also of the form of an f-divergence. For probabilities over a finite alphabet, it is unique in being a member of both of these classes of statistical divergences.
Finance (game theory)
[edit]Consider a growth-optimizing investor in a fair game with mutually exclusive outcomes (e.g. a “horse race” in which the official odds add up to one). The rate of return expected by such an investor is equal to the relative entropy between the investor's believed probabilities and the official odds.[17] This is a special case of a much more general connection between financial returns and divergence measures.[18]
Financial risks are connected to via information geometry.[19] Investors' views, the prevailing market view, and risky scenarios form triangles on the relevant manifold of probability distributions. The shape of the triangles determines key financial risks (both qualitatively and quantitatively). For instance, obtuse triangles in which investors' views and risk scenarios appear on “opposite sides” relative to the market describe negative risks, acute triangles describe positive exposure, and the right-angled situation in the middle corresponds to zero risk.
Motivation
[edit]![](http://upload.wikimedia.org/wikipedia/commons/thumb/a/a8/KL-Gauss-Example.png/320px-KL-Gauss-Example.png)
In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value out of a set of possibilities X can be seen as representing an implicit probability distribution over X, where is the length of the code for in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution Q is used, compared to using a code based on the true distribution P: it is the excess entropy.
where is the cross entropy of P and Q, and is the entropy of P (which is the same as the cross-entropy of P with itself).
The relative entropy can be thought of geometrically as a statistical distance, a measure of how far the distribution Q is from the distribution P. Geometrically it is a divergence: an asymmetric, generalized form of squared distance. The cross-entropy is itself such a measurement (formally a loss function), but it cannot be thought of as a distance, since is not zero. This can be fixed by subtracting to make agree more closely with our notion of distance, as the excess loss. The resulting function is asymmetric, and while this can be symmetrized (see § Symmetrised divergence), the asymmetric form is more useful. See § Interpretations for more on the geometric interpretation.
Relative entropy relates to "rate function" in the theory of large deviations.[20][21]
Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used characterization of entropy.[22] Consequently, mutual information is the only measure of mutual dependence that obeys certain related conditions, since it can be defined in terms of Kullback–Leibler divergence.
Properties
[edit]- Relative entropy is always non-negative, a result known as Gibbs' inequality, with equals zero if and only if as measures.
In particular, if and , then -almost everywhere. The entropy thus sets a minimum value for the cross-entropy , the expected number of bits required when using a code based on Q rather than P; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value x drawn from X, if a code is used corresponding to the probability distribution Q, rather than the "true" distribution P.
- No upper-bound exists for the general case. However, it is shown that if P and Q are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of can be calculated.[23]
- Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under parameter transformations. For example, if a transformation is made from variable x to variable , then, since and where is the absolute value of the derivative or more generally of the Jacobian, the relative entropy may be rewritten: where and . Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a dimensionally consistent quantity, since if x is a dimensioned variable, and are also dimensioned, since e.g. is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory[24] (such as self-information or Shannon entropy), which can become undefined or negative for non-discrete probabilities.
- Relative entropy is additive for independent distributions in much the same way as Shannon entropy. If are independent distributions, and , and likewise for independent distributions then
- Relative entropy is convex in the pair of probability measures , i.e. if and are two pairs of probability measures then
- may be Taylor expanded about its minimum (i.e. ) as which converges if and only if almost surely w.r.t .
Denote and note that . The first derivative of may be derived and evaluated as follows Further derivatives may be derived and evaluated as follows Hence solving for via the Taylor expansion of about evaluated at yields a.s. is a sufficient condition for convergence of the series by the following absolute convergence argument a.s. is also a necessary condition for convergence of the series by the following proof by contradiction. Assume that with measure strictly greater than . It then follows that there must exist some values , , and such that and with measure . The previous proof of sufficiency demonstrated that the measure component of the series where is bounded, so we need only concern ourselves with the behavior of the measure component of the series where . The absolute value of the th term of this component of the series is then lower bounded by , which is unbounded as , so the series diverges.
Duality formula for variational inference
[edit]The following result, due to Donsker and Varadhan,[25] is known as Donsker and Varadhan's variational formula.
Theorem [Duality Formula for Variational Inference] — Let be a set endowed with an appropriate -field , and two probability measures P and Q, which formulate two probability spaces and , with . ( indicates that Q is absolutely continuous with respect to P.) Let h be a real-valued integrable random variable on . Then the following equality holds
Further, the supremum on the right-hand side is attained if and only if it holds
almost surely with respect to probability measure P, where denotes the Radon-Nikodym derivative of Q with respect to P .
For a short proof assuming integrability of with respect to P, let have P-density , i.e. Then
Therefore,
where the last inequality follows from , for which equality occurs if and only if . The conclusion follows.
For alternative proof using measure theory, see.[26]
Examples
[edit]Multivariate normal distributions
[edit]Suppose that we have two multivariate normal distributions, with means and with (non-singular) covariance matrices If the two distributions have the same dimension, k, then the relative entropy between the distributions is as follows:[27]
The logarithm in the last term must be taken to base e since all terms apart from the last are base-e logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in nats. Dividing the entire expression above by yields the divergence in bits.
In a numerical implementation, it is helpful to express the result in terms of the Cholesky decompositions such that and . Then with M and y solutions to the triangular linear systems , and ,
A special case, and a common quantity in variational inference, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):
For two univariate normal distributions p and q the above simplifies to[28]
In the case of co-centered normal distributions with , this simplifies[29] to:
Uniform distributions
[edit]Consider two uniform distributions, with the support of enclosed within (). Then the information gain is:
Intuitively,[29] the information gain to a k times narrower uniform distribution contains bits. This connects with the use of bits in computing, where bits would be needed to identify one element of a k long stream.
Relation to metrics
[edit]While relative entropy is a statistical distance, it is not a metric on the space of probability distributions, but instead it is a divergence.[4] While metrics are symmetric and generalize linear distance, satisfying the triangle inequality, divergences are asymmetric in general and generalize squared distance, in some cases satisfying a generalized Pythagorean theorem. In general does not equal , and while this can be symmetrized (see § Symmetrised divergence), the asymmetry is an important part of the geometry.[4]
It generates a topology on the space of probability distributions. More concretely, if is a sequence of distributions such that
- ,
then it is said that
- .
Pinsker's inequality entails that
- ,
where the latter stands for the usual convergence in total variation.
Fisher information metric
[edit]Relative entropy is directly related to the Fisher information metric. This can be made explicit as follows. Assume that the probability distributions P and Q are both parameterized by some (possibly multi-dimensional) parameter . Consider then two close by values of and so that the parameter differs by only a small amount from the parameter value . Specifically, up to first order one has (using the Einstein summation convention)
with a small change of in the j direction, and the corresponding rate of change in the probability distribution. Since relative entropy has an absolute minimum 0 for , i.e. , it changes only to second order in the small parameters . More formally, as for any minimum, the first derivatives of the divergence vanish
and by the Taylor expansion one has up to second order
where the Hessian matrix of the divergence
must be positive semidefinite. Letting vary (and dropping the subindex 0) the Hessian defines a (possibly degenerate) Riemannian metric on the θ parameter space, called the Fisher information metric.
Fisher information metric theorem
[edit]When satisfies the following regularity conditions:
- exist,
where ξ is independent of ρ
then:
Variation of information
[edit]Another information-theoretic metric is variation of information, which is roughly a symmetrization of conditional entropy. It is a metric on the set of partitions of a discrete probability space.
Relation to other quantities of information theory
[edit]Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.
Self-information
[edit]The self-information, also known as the information content of a signal, random variable, or event is defined as the negative logarithm of the probability of the given outcome occurring.
When applied to a discrete random variable, the self-information can be represented as[citation needed]
is the relative entropy of the probability distribution from a Kronecker delta representing certainty that — i.e. the number of extra bits that must be transmitted to identify i if only the probability distribution is available to the receiver, not the fact that .
Mutual information
[edit]The mutual information,
is the relative entropy of the joint probability distribution from the product of the two marginal probability distributions — i.e. the expected number of extra bits that must be transmitted to identify X and Y if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability is known, it is the expected number of extra bits that must on average be sent to identify Y if the value of X is not already known to the receiver.
Shannon entropy
[edit]The Shannon entropy,
is the number of bits which would have to be transmitted to identify X from N equally likely possibilities, less the relative entropy of the uniform distribution on the random variates of X, , from the true distribution — i.e. less the expected number of bits saved, which would have had to be sent if the value of X were coded according to the uniform distribution rather than the true distribution . This definition of Shannon entropy forms the basis of E.T. Jaynes's alternative generalization to continuous distributions, the limiting density of discrete points (as opposed to the usual differential entropy), which defines the continuous entropy as
which is equivalent to:
Conditional entropy
[edit]is the number of bits which would have to be transmitted to identify X from N equally likely possibilities, less the relative entropy of the product distribution from the true joint distribution — i.e. less the expected number of bits saved which would have had to be sent if the value of X were coded according to the uniform distribution rather than the conditional distribution of X given Y.
Cross entropy
[edit]When we have a set of possible events, coming from the distribution p, we can encode them (with a lossless data compression) using entropy encoding. This compresses the data by replacing each fixed-length input symbol with a corresponding unique, variable-length, prefix-free code (e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be encoded as the bits (0, 10, 11)). If we know the distribution p in advance, we can devise an encoding that would be optimal (e.g.: using Huffman coding). Meaning the messages we encode will have the shortest length on average (assuming the encoded events are sampled from p), which will be equal to Shannon's Entropy of p (denoted as ). However, if we use a different probability distribution (q) when creating the entropy encoding scheme, then a larger number of bits will be used (on average) to identify an event from a set of possibilities. This new (larger) number is measured by the cross entropy between p and q.
The cross entropy between two probability distributions (p and q) measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p. The cross entropy for two distributions p and q over the same probability space is thus defined as follows.
For explicit derivation of this, see the Motivation section above.
Under this scenario, relative entropies (kl-divergence) can be interpreted as the extra number of bits, on average, that are needed (beyond ) for encoding the events because of using q for constructing the encoding scheme instead of p.
Bayesian updating
[edit]In Bayesian statistics, relative entropy can be used as a measure of the information gain in moving from a prior distribution to a posterior distribution: . If some new fact is discovered, it can be used to update the posterior distribution for X from to a new posterior distribution using Bayes' theorem:
This distribution has a new entropy:
which may be less than or greater than the original entropy . However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on instead of a new code based on would have added an expected number of bits:
to the message length. This therefore represents the amount of useful information, or information gain, about X, that has been learned by discovering .
If a further piece of data, , subsequently comes in, the probability distribution for x can be updated further, to give a new best guess . If one reinvestigates the information gain for using rather than , it turns out that it may be either greater or less than previously estimated:
- may be ≤ or > than
and so the combined information gain does not obey the triangle inequality:
- may be <, = or > than
All one can say is that on average, averaging using , the two sides will average out.
Bayesian experimental design
[edit]A common goal in Bayesian experimental design is to maximise the expected relative entropy between the prior and the posterior.[31] When posteriors are approximated to be Gaussian distributions, a design maximising the expected relative entropy is called Bayes d-optimal.
Discrimination information
[edit]Relative entropy can also be interpreted as the expected discrimination information for over : the mean information per sample for discriminating in favor of a hypothesis against a hypothesis , when hypothesis is true.[32] Another name for this quantity, given to it by I. J. Good, is the expected weight of evidence for over to be expected from each sample.
The expected weight of evidence for over is not the same as the information gain expected per sample about the probability distribution of the hypotheses,
Either of the two quantities can be used as a utility function in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.
On the entropy scale of information gain there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the logit scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the Riemann hypothesis is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of loss function for uncertainty are both useful, according to how well each reflects the particular circumstances of the problem in question.
Principle of minimum discrimination information
[edit]The idea of relative entropy as discrimination information led Kullback to propose the Principle of Minimum Discrimination Information (MDI): given new facts, a new distribution f should be chosen which is as hard to discriminate from the original distribution as possible; so that the new data produces as small an information gain as possible.
For example, if one had a prior distribution over x and a, and subsequently learnt the true distribution of a was , then the relative entropy between the new joint distribution for x and a, , and the earlier prior distribution would be:
i.e. the sum of the relative entropy of the prior distribution for a from the updated distribution , plus the expected value (using the probability distribution ) of the relative entropy of the prior conditional distribution from the new conditional distribution . (Note that often the later expected value is called the conditional relative entropy (or conditional Kullback–Leibler divergence) and denoted by [3][30]) This is minimized if over the whole support of ; and we note that this result incorporates Bayes' theorem, if the new distribution is in fact a δ function representing certainty that a has one particular value.
MDI can be seen as an extension of Laplace's Principle of Insufficient Reason, and the Principle of Maximum Entropy of E.T. Jaynes. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see differential entropy), but the relative entropy continues to be just as relevant.
In the engineering literature, MDI is sometimes called the Principle of Minimum Cross-Entropy (MCE) or Minxent for short. Minimising relative entropy from m to p with respect to m is equivalent to minimizing the cross-entropy of p and m, since
which is appropriate if one is trying to choose an adequate approximation to p. However, this is just as often not the task one is trying to achieve. Instead, just as often it is m that is some fixed prior reference measure, and p that one is attempting to optimise by minimising subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be