PAC-learning is Undecidable

The paper demonstrates that falsifiability is fundamental to learning. We prove the following theorem for statistical learning and sequential prediction: If a theory is falsifiable then it is learnable -- i.e. admits a strategy that predicts optimally. An analogous result is shown for universal induction.

Lecture Notes in Computer Science, 1995

We prove that log n-decision lists |the class of decision lists such that all their terms have low Kolmogorov complexity| are learnable in the simple PAC learning model. The proof is based on a transformation from an algorithm based on equivalence queries (found independently by Simon). Then we introduce the class of simple decision lists, and extend our algorithm to show that simple decision lists are simple-PAC learnable as well. This last result is relevant in that it is, to our knowledge, the rst learning algorithm for decision lists in which an exponentially wide set of functions may be used for the terms.

There is a strong consensus that combining the versatility of machine learning with the assurances given by formal verification is highly desirable. It is much less clear what verified machine learning should mean exactly. We consider this question from the (unexpected?) perspective of computable analysis. This allows us to define the computational tasks underlying verified ML in a model-agnostic way, and show that they are in principle computable.

Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2012

Initially discussed are some of Alan Turing's wonderfully profound and influential ideas about mind and mechanism—including regarding their connection to the main topic of the present study, which is within the field of computability-theoretic learning theory. Herein is investigated the part of this field concerned with the algorithmic, trial-and-error inference of eventually correct programs for functions from their data points. As to the main content of this study: in prior papers, beginning with the seminal work by Freivalds et al. in 1995, the notion of intrinsic complexity is used to analyse the learning complexity of sets of functions in a Gold-style learning setting. Herein are pointed out some weaknesses of this notion. Offered is an alternative based on epitomizing sets of functions—sets that are learnable under a given learning criterion, but not under other criteria that are not at least as powerful. To capture the idea of epitomizing sets, new reducibility notions ar...

Journal of Computational Science, 2024

with Vit Stritecky. Despite an increasing role of machine learning in science, there is a lack of results on limits of empirical exploration aided by machine learning. In this paper, we construct one such limit by proving undecidability of learnability of state spaces of physical systems. We characterize state spaces as binary hypothesis classes of the computable Probably Approximately Correct learning framework. This leads to identifying the first limit for learnability of state spaces in the agnostic setting. Further, using the fact that finiteness of the combinatorial dimension of hypothesis classes is undecidable, we derive undecidability for learnability of state spaces as well. Throughout the paper, we try to connect our formal results with modern neural networks. This allows us to bring the limits close to the current practice and make a first step in connecting scientific exploration aided by machine learning with results from learning theory.

2009

Theorem 1. Let d be the VC dimension of concept space C, and let Vn={h∈ C:∀ i≤ n, h (xi)= f (xi)}, where f∈ C is the target function (ie, er (f)= 0), and (x1, x2,..., xn)∼ Dn is a sequence of iid training examples. Then for any δ∈(0, 1), with probability≥ 1− δ,∀ h∈ Vn, er (h)≤ 24 n (d ln (880θf)+ ln 12 δ).(3)

arXiv (Cornell University), 2022

Journal of Computational Science, 2024

Despite an increasing role of machine learning in science, there is a lack of results on limits of empirical exploration aided by machine learning. In this paper, we construct one such limit by proving undecidability of learnability of state spaces of physical systems. We characterize state spaces as binary hypothesis classes of the computable Probably Approximately Correct learning framework. This leads to identifying the first limit for learnability of state spaces in the agnostic setting. Further, using the fact that finiteness of the combinatorial dimension of hypothesis classes is undecidable, we derive undecidability for learnability of state spaces as well. Throughout the paper, we try to connect our formal results with modern neural networks. This allows us to bring the limits close to the current practice and make a first step in connecting scientific exploration aided by machine learning with results from learning theory.

Theoretical Computer Science, 1998

This paper focuses on a general setup for obtaining sample size lower bounds for learning concept classes under fixed distribution laws in an extended PAC learning framework. These bounds do not depend on the running time of learning procedures and are information-theoretic in nature. They are based on incompressibility methods drawn from Kolmogorov Complexity and Algorithmic Probability theories.

arXiv (Cornell University), 2013

We define a class of functions termed "Computable in the Limit", based on the Machine Learning paradigm of "Identification in the Limit". A function is Computable in the Limit if it defines a property Pp of a recursively enumerable class A of recursively enumerable data sequences S ∈ A, such that each data sequence S is generated by a total recursive function φs that enumerates S. Let the index s represent the data sequence S. The property Pp(s) = x is computed by a partial recursive function φp(s, t) such that there exists a u where φp(s, u) = x and for all t ≥ u, φp(s, t) = x if it converges. Since the index s is known, this is not an identification problem -instead it is computing a common property of the sequences in A. We give a Normal Form Theorem for properties that are Computable in the Limit, similar to Kleene's Normal Form Theorem. We also give some examples of sets that are Computable in the Limit, and derive some properties of Canonical and Complexity Bound Enumerations of classes of total functions, and show that no full enumeration of all indices of Turing machines T M i that compute a given total function f (x) can be Computable in the Limit.