Surprisal analysis

Surprisal analysis is an information-theoretical analysis technique that integrates and applies principles of thermodynamics and maximal entropy. Surprisal analysis is capable of relating the underlying microscopic properties to the macroscopic bulk properties of a system. It has already been applied to a spectrum of disciplines including engineering, physics, chemistry and biomedical engineering. Recently, it has been extended to characterize the state of living cells, specifically monitoring and characterizing biological processes in real time using transcriptional data.

Surprisal analysis was formulated at the Hebrew University of Jerusalem as a joint effort between Raphael David Levine, Richard Barry Bernstein and Avinoam Ben-Shaul in 1972. Levine and colleagues had recognized a need to better understand the dynamics of non-equilibrium systems, particularly of small systems, that are not seemingly applicable to thermodynamic reasoning.^[1] Alhassid and Levine first applied surprisal analysis in nuclear physics, to characterize the distribution of products in heavy ion reactions. Since its formulation, surprisal analysis has become a critical tool for the analysis of reaction dynamics and is an official IUPAC term.^[2]*

Maximum entropy methods are at the core of a new view of scientific inference, allowing analysis and interpretation of large and sometimes noisy data. Surprisal analysis extends principles of maximal entropy and of thermodynamics, where both equilibrium thermodynamics and statistical mechanics are assumed to be inferential processes. This enables surprisal analysis to be an effective method of information quantification and compaction and of providing an unbiased characterization of systems. Surprisal analysis is particularly useful to characterize and understand dynamics in small systems, where energy fluxes that are otherwise negligible in large systems heavily influence system behavior.

Foremost, surprisal analysis identifies the state of a system when it reaches its maximal entropy, or thermodynamic equilibrium. This is known as balance state of the system because once a system reaches its maximal entropy, it can no longer initiate or participate in spontaneous processes. Following the determination of the balanced state, surprisal analysis then characterizes all the states in which the system deviates away from the balance state. These deviations are caused by constraints; these constraints on the system prevent the system from reaching its maximal entropy. Surprisal analysis is applied to both identify and characterize these constraints. In terms of the constraints, the probability ${\displaystyle P(n)}$ of an event ${\displaystyle n}$ is quantified by

${\displaystyle P(n)=P^{0}(n)\exp \left[-\sum _{\alpha }\lambda _{\alpha }G_{\alpha }(n)\right]}$.

Here ${\displaystyle P^{0}(n)}$ is the probability of the event ${\displaystyle n}$ in the balanced state. It is usually called the “prior probability” because it is the probability of an event ${\displaystyle n}$ prior to any constraints. The surprisal itself is defined as

${\displaystyle {\begin{aligned}{\text{surprisal}}&{\stackrel {\text{def}}{=}}-\ln {\frac {P(n)}{P^{0}(n)}}\\&=\sum _{\alpha }\lambda _{\alpha }G_{\alpha }(n)\end{aligned}}}$

The surprisal equals the sum over the constraints and is a measure of the deviation from the balanced state. These deviations are ranked on the degree of deviation from the balance state and ordered on the most to least influential to the system. This ranking is provided through the use of Lagrange multipliers. The most important constraint and usually the constraint sufficient to characterize a system exhibit the largest Lagrange multiplier. The multiplier for constraint ${\displaystyle \alpha }$ is denoted above as ${\displaystyle \lambda _{\alpha }}$; larger multipliers identify more influential constraints. The event variable ${\displaystyle G_{\alpha }(n)}$ is the value of the constraint ${\displaystyle \alpha }$ for the event ${\displaystyle n}$. Using the method of Lagrange multipliers^[3] requires that the prior probability ${\displaystyle P^{0}(n)}$ and the nature of the constraints be experimentally identified. A numerical algorithm for determining Lagrange multipliers has been introduced by Agmon et al.^[4] Recently, singular value decomposition and principal component analysis of the surprisal was utilized to identify constraints on biological systems, extending surprisal analysis to better understanding biological dynamics as shown in the figure.

Surprisal (a term coined^[5] in this context by Myron Tribus^[6]) was first introduced to better understand the specificity of energy release and selectivity of energy requirements of elementary chemical reactions.^[1] This gave rise to a series of new experiments which demonstrated that in elementary reactions, the nascent products could be probed and that the energy is preferentially released and not statistically distributed.^[1] Surprisal analysis was initially applied to characterize a small three molecule system that did not seemingly conform to principles of thermodynamics and a single dominant constraint was identified that was sufficient to describe the dynamic behavior of the three molecule system. Similar results were then observed in nuclear reactions, where differential states with varying energy partitioning are possible. Often chemical reactions require energy to overcome an activation barrier. Surprisal analysis is applicable to such applications as well.^[7] Later, surprisal analysis was extended to mesoscopic systems, bulk systems ^[3] and to dynamical processes.^[8]

Surprisal analysis was extended to better characterize and understand cellular processes,^[9] see figure, biological phenomena and human disease with reference to personalized diagnostics. Surprisal analysis was first utilized to identify genes implicated in the balance state of cells in vitro; the genes mostly present in the balance state were genes directly responsible for the maintenance of cellular homeostasis.^[10] Similarly, it has been used to discern two distinct phenotypes during the EMT of cancer cells.^[11]

• Information content
• Information theory
• Singular value decomposition
• Principal component analysis
• Entropy
• Decision tree learning
• Information gain in decision trees