On ordinal VC-dimension and some notions of complexity

2000

We evaluate the required size of a possible proof of Arrow's impossibility theorem in a proof-theoretic manner. The primary purpose is the consideration of (contentwise) complexity required for some statements, and Arrow's theorem is taken as an instance in order to show the possibility that even in a finite case, the required proof is inevitably gigantic. We consider the simplest case with two individuals, three social alternatives, and linear orderings for individual and social preferences. We formulate Arrow's theorem in propositional classical logic in the Gentzen-style proof theory. The size of a proof is measured by the number of leaves of a proof tree. We show that a proof is necessarily gigantic; a lower bound is 6 36 and an upper bound 6 37 + 420. These numbers exceed the limit of human manageability to construct such a proof, but we have a proof of Arrow's theorem, which appears contradictory. We discuss this result from various points of views such as deductive and inductive game theoretical points of view. * The authors thank Tai-Wei Hu and Ken Urai for helpful comments on an earlier draft of the paper.

2004

Freivalds de ned an acceptable programming system independent criterion for learning programs for functions in which the nal programs were required to be both correct and \nearly" minimal size, i.e, within a computable function of being purely minimal size. Kinber showed that this parsimony requirement on nal programs limits learning power. However, in scienti c inference, parsimony is considered highly desirable. A lim-computable function is (by de nition) one calculable by a total procedure allowed to change its mind nitely many times about its output. Investigated is the possibility of assuaging somewhat the limitation on learning power resulting from requiring parsimonious nal programs by use of criteria which require the nal, correct programs to be \not-so-nearly" minimal size, e.g., to be within a lim-computable function of actual minimal size. It is shown that some parsimony in the nal program is thereby retained, yet learning power strictly increases. Considered, then, are lim-computable functions as above but for which notations for constructive ordinals are used to bound the number of mind changes allowed regarding the output. This is a variant of an idea introduced by Freivalds and Smith. For this ordinal notation complexity bounded version of lim-computability, the power of the resultant learning criteria form nely graded, in nitely ramifying, in nite hierarchies intermediate between the computable and the lim-computable cases. Some of these hierarchies, for the natural notations determining them, are shown to be optimally tight.

2020

We initiate a study of learning with computable learners and computable output predictors. Recent results in statistical learning theory have shown that there are basic learning problems whose learnability can not be determined within ZFC (Ben-David et al. (2017, 2019)). This motivates us to consider learnability by algorithms with computable output predictors (both learners and predictors are then representable as finite objects). We thus propose the notion of CPAC learnability, by adding some basic computability requirements into a PAC learning framework. As a first step towards a characterization, we show that in this framework learnability of a binary hypothesis class is not implied by finiteness of its VC-dimension anymore. We also present some situations where we are guaranteed to have a computable learner.

Information Processing Letters, 1997

This paper focuses on the relation between computational learning theory and resource-bounded dimension. We intend to establish close connections between the learnability/nonlearnability of a concept class and its corresponding size in terms of effective dimension, which will allow the use of powerful dimension techniques in computational learning and viceversa, the import of learning results into complexity via dimension. Firstly, we obtain a tight result on the dimension of online mistake-bound learnable classes. Secondly, in relation with PAC learning, we show that the polynomial-space dimension of PAC learnable classes of concepts is zero. This provides a hypothesis on effective dimension that implies the inherent unpredictability of concept classes (the classes that verify this property are classes not efficiently PAC learnable using any hypothesis). Thirdly, in relation to space dimension of classes that are learnable by membership query algorithms, the main result proves that polynomial-space dimension of concept classes learnable by a membership-query algorithm is zero.

Journal of Computer and System Sciences, 2008

We consider the complexity of properly learning concept classes, i.e. when the learner must output a hypothesis of the same form as the unknown concept. We present the following new upper and lower bounds on well-known concept classes:

2009

Model theory is a branch of mathematical logic that investigates the logical properties of mathematical structures. It has been quite successfully applied to computational complexity resulting in an area of research called descriptive complexity theory. Descriptive complexity is essentially a syntactical characterization of complexity classes using logical formalisms. However, there are still much more of model theory technologies that have not yet been explored by complexity theorists, especially the subarea of classification/stability theory. This paper is divided into two parts. The first part quickly surveys the main results of descriptive complexity theory. In the second part we introduce the field of classification/stability theory, then give the outlines of a research project whose aim is to apply this theory to give a semantical characterization of complexity classes. This would initiate a brand new research area.

Springer eBooks, 2004

The two most important notions of fractal dimension are Hausdorff dimension, developed by , and packing dimension, developed independently by and . Both dimensions have the mathematical advantage of being defined from measures, and both have yielded extensive applications in fractal geometry and dynamical systems. has recently proven a simple characterization of Hausdorff dimension in terms of gales, which are betting strategies that generalize martingales. Imposing various computability and complexity constraints on these gales produces a spectrum of effective versions of Hausdorff dimension, including constructive, computable, polynomial-space, polynomial-time, and finite-state dimensions. Work by several investigators has already used these effective dimensions to shed significant new light on a variety of topics in theoretical computer science. In this paper we show that packing dimension can also be characterized in terms of gales. Moreover, even though the usual definition of packing dimension is considerably more complex than that of Hausdorff dimension, our gale characterization of packing dimension is an exact dual of -and every bit as simple as -the gale characterization of Hausdorff dimension. Effectivizing our gale characterization of packing dimension produces a variety of effective strong dimensions, which are exact duals of the effective dimensions mentioned above. In general (and in analogy with the classical fractal dimensions), the effective strong dimension of a set or sequence is at least as great as its effective dimension, with equality for sets or sequences that are sufficiently regular. We develop the basic properties of effective strong dimensions and prove a number of results relating them to fundamental aspects of randomness, Kolmogorov complexity, prediction, Boolean circuit-size complexity, polynomial-time degrees, and data compression. Aside from the above characterization of packing dimension, our two main theorems are the following. 1. If β = (β 0 , β 1 , . . .) is a computable sequence of biases that are bounded away from 0 and R is random with respect to β, then the dimension and strong dimension of R are the lower and upper average entropies, respectively, of β. 2. For each pair of ∆ 0 2 -computable real numbers 0 < α ≤ β ≤ 1, there exists A ∈ E such that the polynomial-time many-one degree of A has dimension α in E and strong dimension β in E. Our proofs of these theorems use a new large deviation theorem for self-information with respect to a bias sequence β that need not be convergent.

Machine Learning, 1994

A central problem in inductive logic programming is theory evaluation. Without some sort of preference criterion, any two theories that explain a set of examples are equally acceptable. This paper presents a scheme for evaluating alternative inductive theories based on an objective preference criterion. It strives to extract maximal redundancy from examples, transforming structure into randomness. A major strength of the method is its application to learning problems where negative examples of concepts are scarce or unavailable.

1997

Gold introduced the notion of learning in the limit where a class S is learnable iff there is a recursive machine M which reads the course of values of a function f and converges to a program for f whenever f is in S. An important measure for the speed of convergence in this model is the quantity of mind changes before the onset of convergence. The oldest model is to consider a constant bound on the number of mind changes M makes on any input function; such a bound is referred here as type 1. Later this was generalized to a bound of type 2 where a counter ranges over constructive ordinals and is counted down at every mind change. Although ordinal bounds permit the inference of richer concept classes than constant bounds, they still are a severe restriction. Therefore the present work introduces two more general approaches to bounding mind changes. These are based on counting by going down in a linearly ordered set (type 3) and on counting by going down in a partially ordered set (type 4). In both cases the set must not contain infinite descending recursive sequences. These four types of mind changes yield a hierarchy and there are identifiable classes that cannot be learned with the most general mind change bound of type 4. It is shown that existence of type 2 bound is equivalent to the existence of a learning algorithm which converges on every (also nonrecursive) input function and the existence of type 4 is shown to be equivalent to the existence of a learning algorithm which converges on every recursive function. A partial characterization of type 3 yields a result of A. Sharma et al. / Information and Computation 189 (2004) independent interest in recursion theory. The interplay between mind change complexity and choice of hypothesis space is investigated. It is established that for certain concept classes, a more expressive hypothesis space can sometimes reduce mind change complexity of learning these classes. The notion of mind change bound for behaviourally correct learning is indirectly addressed by employing the above four types to restrict the number of predictive errors of commission in finite error next value learning (NV ) -a model equivalent to behaviourally correct learning. Again, natural characterizations for type 2 and type 4 bounds are derived. Their naturalness is further illustrated by characterizing them in terms of branches of uniformly recursive families of binary trees. Crown