Around Kolmogorov Complexity: Basic Notions and Results

2013

The famous Gödel incompleteness theorem states that for every consistent sufficiently rich formal theory T there exist true statements that are unprovable in T . Such statements would be natural candidates for being added as axioms, but how can we obtain them? One classical (and well studied) approach is to add to some theory T an axiom that claims the consistency of T . In this paper we discuss another approach motivated by Chaitin's version of Gödel's theorem where axioms claiming the randomness (or incompressibility) of some strings are probabilistically added, and show that it is not really useful, in the sense that this does not help us to prove new interesting theorems. This result (cf. [She06]) answers a question recently asked by Lipton [LR11]. The situation changes if we take into account the size of the proofs: randomly chosen axioms may help making proofs much shorter (unless NP=PSPACE). This result partially answers the question asked in [She06]. We then study the axiomatic power of the statements of type "the Kolmogorov complexity of x exceeds n" (where x is some string, and n is some integer) in general. They are Π1 (universally quantified) statements of Peano arithmetic. We show (Theorem 5) that by adding all true statements of this type, we obtain a theory that proves all true Π1-statements, and also provide a more detailed classification. In particular, as Theorem 7 shows, to derive all true Π1-statements it is enough to add one statement of this type for each n (or even for infinitely many n) if strings are chosen in a special way. On the other hand, one may add statements of this type for most x of length n (for every n) and still obtain a weak theory (Theorem 10). We also study other logical questions related to "random axioms" (hierarchy with respect to n, Theorem 8 in Section 3.3, independence in Section 3.6, etc.). Finally, we consider a theory that claims Martin-Löf randomness of a given infinite binary sequence. This claim can be formalized in different ways. We show that different formalizations are closely related but not equivalent, and study their properties.

We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's information theory have a common purpose, and where they are fundamentally different. We indicate how recent developments within the theory allow one to formally distinguish between `structural' (meaningful) and `random' information as measured by the Kolmogorov structure function, which leads to a mathematical formalization of Occam's razor in inductive inference. We end by discussing some of the philosophical implications of the theory.

The Computer Journal, 1999

The question why and how probability theory can be applied to the real-world phenomena has been discussed for several centuries. When the algorithmic information theory was created, it became possible to discuss these problems in a more specific way. In particular, Li and Vitányi [6], Rissanen [3], Wallace and Dowe [7] have discussed the connection between Kolmogorov (algorithmic) complexity and minimum description length (minimum message length) principle. In this note we try to point out a few simple observations that (we believe) are worth keeping in mind while discussing these topics.

Springer eBooks, 2004

The two most important notions of fractal dimension are Hausdorff dimension, developed by , and packing dimension, developed independently by and . Both dimensions have the mathematical advantage of being defined from measures, and both have yielded extensive applications in fractal geometry and dynamical systems. has recently proven a simple characterization of Hausdorff dimension in terms of gales, which are betting strategies that generalize martingales. Imposing various computability and complexity constraints on these gales produces a spectrum of effective versions of Hausdorff dimension, including constructive, computable, polynomial-space, polynomial-time, and finite-state dimensions. Work by several investigators has already used these effective dimensions to shed significant new light on a variety of topics in theoretical computer science. In this paper we show that packing dimension can also be characterized in terms of gales. Moreover, even though the usual definition of packing dimension is considerably more complex than that of Hausdorff dimension, our gale characterization of packing dimension is an exact dual of -and every bit as simple as -the gale characterization of Hausdorff dimension. Effectivizing our gale characterization of packing dimension produces a variety of effective strong dimensions, which are exact duals of the effective dimensions mentioned above. In general (and in analogy with the classical fractal dimensions), the effective strong dimension of a set or sequence is at least as great as its effective dimension, with equality for sets or sequences that are sufficiently regular. We develop the basic properties of effective strong dimensions and prove a number of results relating them to fundamental aspects of randomness, Kolmogorov complexity, prediction, Boolean circuit-size complexity, polynomial-time degrees, and data compression. Aside from the above characterization of packing dimension, our two main theorems are the following. 1. If β = (β 0 , β 1 , . . .) is a computable sequence of biases that are bounded away from 0 and R is random with respect to β, then the dimension and strong dimension of R are the lower and upper average entropies, respectively, of β. 2. For each pair of ∆ 0 2 -computable real numbers 0 < α ≤ β ≤ 1, there exists A ∈ E such that the polynomial-time many-one degree of A has dimension α in E and strong dimension β in E. Our proofs of these theorems use a new large deviation theorem for self-information with respect to a bias sequence β that need not be convergent.

The Computer Journal, 1999

We briefly discuss the origins, main ideas and principal applications of the theory of Kolmogorov complexity.

2008

Although information content is invariant up to an additive constant, the range of possible additive constants applicable to programming languages is so large that in practice it plays a major role in the actual evaluation of K(s), the Kolmogorov-Chaitin complexity of a string s. Some attempts have been made to arrive at a framework stable enough for a concrete definition of K, independent of any constant under a programming language, by appealing to the naturalness of the language in question. The aim of this paper is to present an approach to overcome the problem by looking at a set of models of computation converging in output probability distribution such that that naturalness can be inferred, thereby providing a framework for a stable definition of K under the set of convergent models of computation.

2019

Romashchenko and Zimand~\cite{rom-zim:c:mutualinfo} have shown that if we partition the set of pairs $(x,y)$ of $n$-bit strings into combinatorial rectangles, then $I(x:y) \geq I(x:y \mid t(x,y)) - O(\log n)$, where $I$ denotes mutual information in the Kolmogorov complexity sense, and $t(x,y)$ is the rectangle containing $(x,y)$. We observe that this inequality can be extended to coverings with rectangles which may overlap. The new inequality essentially states that in case of a covering with combinatorial rectangles, $I(x:y) \geq I(x:y \mid t(x,y)) - \log \rho - O(\log n)$, where $t(x,y)$ is any rectangle containing $(x,y)$ and $\rho$ is the thickness of the covering, which is the maximum number of rectangles that overlap. We discuss applications to communication complexity of protocols that are nondeterministic, or randomized, or Arthur-Merlin, and also to the information complexity of interactive protocols.

Theory of Computing Systems, 2012

This special issue of Theory or Computing Systems consists of papers associated with the 6th Annual CCR conference held in beautiful Cape Town, 31st January-February 4th 2011. The conference series is devoted to issues around algorithmic information theory, Kolmogorov complexity, and their relationship with computability theory, complexity theory, logic and reverse mathematics. In 2011, the conference was co-located with the 8th Annual Computability and Analysis conference. This co-location reflects the increasing interactions between computable analysis and algorithmic randomness through effective analysis of almost everywhere behavior in analysis such as the Lebesgue Differentiation Theorem and things like Brownian motion. Both conferences were admirably overseen by Vasco Brattka and his program committees. The last 10-15 years has seen huge progress in the areas represented by CCR with two long monographs in the area, recognition at the International Congress of Mathematicians, and a thoroughly thriving research community. Whilst the CCR conference uses a model which does not have an actual proceedings at the meeting, the papers in this special issue reflect the meeting's business. These papers reflect the high caliber of both the researchers and the intellectual merit of the investigations. The work spans issues from applications in biology, computer science to fundamental issues about Kolmogorov complexity. The papers all went through the full Theory of Computing Systems journal reviewing process and several of the initial submissions were rejected. Enjoy.

2020

There is a parallelism between Shannon information theory and algorithmic information theory. In particular, the same linear inequalities are true for Shannon entropies of tuples of random variables and Kolmogorov complexities of tuples of strings (Hammer et al., 1997), as well as for sizes of subgroups and projections of sets (Chan, Yeung, Romashchenko, Shen, Vereshchagin, 1998--2002). This parallelism started with the Kolmogorov-Levin formula (1968) for the complexity of pairs of strings with logarithmic precision. Longpre (1986) proved a version of this formula for space-bounded complexities. In this paper we prove an improved version of Longpre&#39;s result with a tighter space bound, using Sipser&#39;s trick (1980). Then, using this space bound, we show that every linear inequality that is true for complexities or entropies, is also true for space-bounded Kolmogorov complexities with a polynomial space overhead.

2019

Consider a bit string x of length n and Kolmogorov complexity αn (for some α &lt; 1). It is always possible to increase the complexity of x by changing a small fraction of bits in x [2]. What happens with the complexity of x when we randomly change each bit independently with some probability τ ? We prove that a linear increase in complexity happens with high probability, but this increase is smaller than in the case of arbitrary change considered in [2]. The amount of the increase depends on x (strings of the same complexity could behave differently). We give exact lower and upper bounds for this increase (with o(n) precision). The same technique is used to prove the results about the (effective Hausdorff) dimension of infinite sequences. We show that random change increases the dimension with probability 1, and provide an optimal lower bound for the dimension of the changed sequence. We also improve a result from [5] and show that for every sequence ω of dimension α there exists a...