Algorithmic probability

In algorithmic information theory, algorithmic probability, also known as Solomonoff probability, is a mathematical method of assigning a prior probability to a given observation. It was invented by Ray Solomonoff in the 1960s. It is used in inductive inference theory and analyses of algorithms. In his general theory of inductive inference, Solomonoff uses the method together with Bayes' rule to obtain probabilities of prediction for an algorithm's future outputs.

From observer states to physics via algorithmic probability

In the mathematical formalism used, the observations have the form of finite binary strings viewed as outputs of Turing machines, and the universal prior is a probability distribution over the set of finite binary strings calculated from a probability distribution over programs (that is, inputs to a universal Turing machine). The prior is universal in the Turing-computability sense, i.e. no string has zero probability. It is not computable, but it can be approximated.

Formally, the probability $P$ is not a probability and it is not computable. It is only "lower semi-computable" and a "semi-measure". By "semi-measure", it means that $0\leq \sum _{x}P(x)<1$. That is, the "probability" does not actually sum up to one, unlike actual probabilities. This is because some inputs to the Turing machine causes it to never halt, which means the probability mass allocated to those inputs is lost. By "lower semi-computable", it means there is a Turing machine that, given an input string $x$, can print out a sequence $y_{1}<y_{2}<\cdots$ that converges to $P(x)$ from below, but there is no such Turing machine that does the same from above.

Overview

Algorithmic probability is the main ingredient of Solomonoff's theory of inductive inference, the theory of prediction based on observations; it was invented with the goal of using it for machine learning; given a sequence of symbols, which one will come next? Solomonoff's theory provides an answer that is optimal in a certain sense, although it is incomputable. Unlike, for example, Karl Popper's informal inductive inference theory, Solomonoff's is mathematically rigorous.

Four principal inspirations for Solomonoff's algorithmic probability were: Occam's razor, Epicurus' principle of multiple explanations, modern computing theory (e.g. use of a universal Turing machine) and Bayes’ rule for prediction.

Occam's razor and Epicurus' principle are essentially two different non-mathematical approximations of the universal prior.

• Occam's razor: among the theories that are consistent with the observed phenomena, one should select the simplest theory.
• Epicurus' principle of multiple explanations: if more than one theory is consistent with the observations, keep all such theories.

At the heart of the universal prior is an abstract model of a computer, such as a universal Turing machine. Any abstract computer will do, as long as it is Turing-complete, i.e. every computable function has at least one program that will compute its application on the abstract computer.

The abstract computer is used to give precise meaning to the phrase "simple explanation". In the formalism used, explanations, or theories of phenomena, are computer programs that generate observation strings when run on the abstract computer. Each computer program is assigned a weight corresponding to its length. The universal probability distribution is the probability distribution on all possible output strings with random input, assigning for each finite output prefix q the sum of the probabilities of the programs that compute something starting with q. Thus, a simple explanation is a short computer program. A complex explanation is a long computer program. Simple explanations are more likely, so a high-probability observation string is one generated by a short computer program, or perhaps by any of a large number of slightly longer computer programs. A low-probability observation string is one that can only be generated by a long computer program.

Algorithmic probability is closely related to the concept of Kolmogorov complexity. Kolmogorov's introduction of complexity was motivated by information theory and problems in randomness, while Solomonoff introduced algorithmic complexity for a different reason: inductive reasoning. A single universal prior probability that can be substituted for each actual prior probability in Bayes's rule was invented by Solomonoff with Kolmogorov complexity as a side product. It predicts the most likely continuation of that observation, and provides a measure of how likely this continuation will be.

Solomonoff's enumerable measure is universal in a certain powerful sense, but the computation time can be infinite. One way of dealing with this issue is a variant of Leonid Levin's Search Algorithm, which limits the time spent computing the success of possible programs, with shorter programs given more time. When run for longer and longer periods of time, it will generate a sequence of approximations which converge to the universal probability distribution. Other methods of dealing with the issue include limiting the search space by including training sequences.

Solomonoff proved this distribution to be machine-invariant within a constant factor (called the invariance theorem).

Fundamental Theorems

I. Kolmogorov's Invariance Theorem

Kolmogorov's Invariance theorem clarifies that the Kolmogorov Complexity, or Minimal Description Length, of a dataset is invariant to the choice of Turing-Complete language used to simulate a Universal Turing Machine:

$\forall x\in \{0,1\}^{*},|K_{U}(x)-K_{U'}(x)|\leq {\mathcal {O}}(1)$

where $K_{U}(x)=\min _{p}\{|p|:U(p)=x\}$.

Interpretation

The minimal description $p$ such that $U(p)=x$ serves as a natural representation of the string $x$ relative to the Turing-Complete language $U$. Moreover, as $x$ can't be compressed further $p$ is an incompressible and hence uncomputable string. This corresponds to a scientists' notion of randomness and clarifies the reason why Kolmogorov Complexity is not computable.

It follows that any piece of data has a necessary and sufficient representation in terms of a random string.

Proof

The following is taken from

From the theory of compilers, it is known that for any two Turing-Complete languages $U_{1}$ and $U_{2}$, there exists a compiler $\Lambda _{1}$ expressed in $U_{1}$ that translates programs expressed in $U_{2}$ into functionally-equivalent programs expressed in $U_{1}$.

It follows that if we let $p$ be the shortest program that prints a given string $x$ then:

$K_{U_{1}}(x)\leq |\Lambda _{1}|+|p|\leq K_{U_{2}}(x)+{\mathcal {O}}(1)$

where $|\Lambda _{1}|={\mathcal {O}}(1)$, and by symmetry we obtain the opposite inequality.

II. Levin's Universal Distribution

Given that any uniquely-decodable code satisfies the Kraft-McMillan inequality, prefix-free Kolmogorov Complexity allows us to derive the Universal Distribution:

$P(x)=\sum _{U(p)=x}P(U(p)=x)=\sum _{U(p)=x}2^{-K_{U}(p)}\leq 1$

where the fact that $U$ may simulate a prefix-free UTM implies that for two distinct descriptions $p$ and $p'$, $p$ isn't a substring of $p'$ and $p'$ isn't a substring of $p$.

Interpretation

In a Computable Universe, given a phenomenon with encoding $x\in \{0,1\}^{*}$ generated by a physical process the probability of that phenomenon is well-defined and equal to the sum over the probabilities of distinct and independent causes. The prefix-free criterion is precisely what guarantees causal independence.

Proof

This is an immediate consequence of the Kraft-McMillan inequality.

Kraft's inequality states that given a sequence of strings $\{x_{i}\}_{i=1}^{n}$ there exists a prefix code with codewords $\{\sigma _{i}\}_{i=1}^{n}$ where $\forall i,|\sigma _{i}|=k_{i}$ if and only if:

$\sum _{i=1}^{n}s^{-k_{i}}\leq 1$

where $s$ is the size of the alphabet $S$.

Without loss of generality, let's suppose we may order the $k_{i}$ such that:

$k_{1}\leq k_{2}\leq ...\leq k_{n}$

Now, there exists a prefix code if and only if at each step $j$ there is at least one codeword to choose that does not contain any of the previous $j-1$ codewords as a prefix. Due to the existence of a codeword at a previous step $i<j,s^{k_{j}-k_{i}}$ codewords are forbidden as they contain $\sigma _{i}$ as a prefix. It follows that in general a prefix code exists if and only if:

$\forall j\geq 2,s^{k_{j}}>\sum _{i=1}^{j-1}s^{k_{j}-k_{i}}$

Dividing both sides by $s^{k_{j}}$, we find:

$\sum _{i=1}^{n}s^{-k_{i}}\leq 1$

QED.

History

Solomonoff invented the concept of algorithmic probability with its associated invariance theorem around 1960, publishing a report on it: "A Preliminary Report on a General Theory of Inductive Inference." He clarified these ideas more fully in 1964 with "A Formal Theory of Inductive Inference," Part I and Part II.

Key people

• Ray Solomonoff
• Andrey Kolmogorov
• Leonid Levin

See also

• Solomonoff's theory of inductive inference
• Algorithmic information theory
• Bayesian inference
• Inductive inference
• Inductive probability
• Kolmogorov complexity
• Universal Turing machine
• Information-based complexity

Sources

• Li, M. and Vitanyi, P., An Introduction to Kolmogorov Complexity and Its Applications, 3rd Edition, Springer Science and Business Media, N.Y., 2008
• Hutter, Marcus (2005). Universal artificial intelligence: sequential decisions based on algorithmic probability. Texts in theoretical computer science. Berlin Heidelberg: Springer. ISBN 978-3-540-22139-5.

Further reading

• Rathmanner, S and Hutter, M., "A Philosophical Treatise of Universal Induction" in Entropy 2011, 13, 1076-1136: A very clear philosophical and mathematical analysis of Solomonoff's Theory of Inductive Inference

External links

• Algorithmic Probability at Scholarpedia
• Solomonoff's publications