^BBSnline

TRADING SPACES: COMPUTATION, REPRESENTATION AND THE LIMITS OF UNINFORMED LEARNING

Clark, Andy and Thornton, Chris (1997) TRADING SPACES: COMPUTATION, REPRESENTATION AND THE LIMITS OF UNINFORMED LEARNING.

Short Abstract:

Some regularities enjoy only an attenuated existence in a body of training data. These are regularities whose statistical visibility depends on some systematic re-coding of the data. The space of possible re-codings is, however, infinitely large - it is the space of applicable Turing machines. As a result, mappings which pivot on such attenuated regularities cannot, in general, be found by brute force search. The class of problems which present such mappings we call the class of `type-2 problems'. Type-1 problems, by contrast, present tractable problems of search insofar as the relevant regularities can be found by sampling the input data as originally coded. Type-2 problems, we suggest, present neither rare nor pathological cases. They are rife in biologically realistic settings and in domains ranging from simple animat behaviors to language acquisition. Not only are such problems rife - they are standardly solved! This presents a puzzle. How, given the statistical intractability of these type-2 cases does nature turn the trick? One answer, which we do not pursue, is to suppose that evolution gifts us with exactly the right set of re-coding biases so as to reduce specific type-2 problems to (tractable) type-1 mappings. Such a heavy duty nativism is no doubt sometimes plausible. But we believe there are other, more general mechanisms also at work. Such mechanisms provide general (not task-specific) strategies for managing problems of type-2 complexity. Several such mechanisms are investigated. At the heart of each is a fundamental ploy viz. the maximal exploitation of states of representation already achieved by prior (type-1) learning so as to reduce the amount of subsequent computational search. Such exploitation both characterises and helps make unitary sense of a diverse range of mechanisms. These include simple incremental learning (Elman 1993), modular connectionism (Jacobs, Jordan and Barto 1991), and the developmental hypothesis of `representational redescription' (Karmiloff-Smith A Functional 1979, Karmiloff-Smith PDP 1992). In addition, the most distinctive features of human cognition---language and culture---may themselves be viewed as adaptations enabling this representation/computation trade-off to be pursued on an even grander scale.

Long Abstract:

Some regularities enjoy only an attenuated existence in a body of training data. These are regularities whose statistical visibility depends on some systematic re-coding of the data. The space of possible re-codings is, however, infinitely large - it is the space of applicable Turing machines. As a result, mappings which pivot on such attenuated regularities cannot, in general, be found by brute force search. The class of problems which present such mappings we call the class of `type-2 problems'. Type-1 problems, by contrast, present tractable problems of search insofar as the relevant regularities can be found by sampling the input data as originally coded. Type-2 problems, we suggest, present neither rare nor pathological cases. They are rife in biologically realistic settings and in domains ranging from simple animat behaviors to language acquisition. Not only are such problems rife - they are standardly solved! This presents a puzzle. How, given the statistical intractability of these type-2 cases does nature turn the trick? One answer, which we do not pursue, is to suppose that evolution gifts us with exactly the right set of re-coding biases so as to reduce specific type-2 problems to (tractable) type-1 mappings. Such a heavy duty nativism is no doubt sometimes plausible. But we believe there are other, more general mechanisms also at work. Such mechanisms provide general (not task-specific) strategies for managing problems of type-2 complexity. Several such mechanisms are investigated. At the heart of each is a fundamental ploy viz. the maximal exploitation of states of representation already achieved by prior (type-1) learning so as to reduce the amount of subsequent computational search. Such exploitation both characterises and helps make unitary sense of a diverse range of mechanisms. These include simple incremental learning (Elman 1993), modular connectionism (Jacobs, Jordan and Barto 1991), and the developmental hypothesis of `representational redescription' (Karmiloff-Smith A Functional 1979, Karmiloff-Smith PDP 1992). In addition, the most distinctive features of human cognition---language and culture---may themselves be viewed as adaptations enabling this representation/computation trade-off to be pursued on an even grander scale.

Keywords:	Learning, connectionism, statistics, representation, search
Subjects:	Psychology: Applied Cognitive Psychology Psychology: Cognitive Psychology Neuroscience: Computational Neuroscience Computer Science: Statistical Models Psychology: Learning and Memory
ID code:	bbs00000444
Deposited by:	Andy Clark on 01 May 2001