To be published in Behavioral and Brain Sciences (in press)

Below is the unedited, uncorrected final draft of a BBS target article that has been accepted

 for publication. This preprint has been prepared for potential commentators who wish to

nominate themselves for formal commentary invitation. Please DO NOT write a commentary

 until you receive a formal invitation. If you are invited to submit a commentary, a copyedited,

corrected version of this paper will be posted.

Base-rate Respect: From Ecological Rationality to Dual Processes

Running head: Base-rate respect

Aron K. Barbey

Department of Psychology

Emory University

Atlanta, GA 30322

(404) 727-7386

abarbey@emory.edu

http://www.psychology.emory.edu/cognition/abarbey/index.html

Steven A. Sloman

Cognitive and Linguistics Science

Brown University, Box 1978

Providence, RI 02912

(401) 863-7595

Steven_Sloman@brown.edu

http://www.cog.brown.edu/~sloman/

Abstract: The phenomenon of base-rate neglect has elicited much debate. One arena of debate concerns how people make judgments under conditions of uncertainty. Another more controversial arena concerns human rationality. In this paper, we attempt to unpack the perspectives in the literature on both kinds of issues and evaluate their ability to explain existing data and their conceptual coherence. We will conclude that the best account of the data should be framed in terms of a dual-process model of judgment that attributes base-rate neglect to associative judgment strategies that fail to adequately represent the set structure of the problem. Base-rate neglect is reduced when problems are presented in a format that affords accurate representation in terms of nested sets of individuals.

Keywords: Base-rate neglect, Probability judgment, Bayesian reasoning, Dual process theory, Nested set hypothesis

1.0. Introduction

Diagnosing whether a patient has a disease, predicting whether a defendant is guilty of a crime, and other everyday as well as life-changing decisions in part reflect the decision-maker’s subjective degree of belief in uncertain events. Intuitions about probability frequently deviate dramatically from the dictates of probability theory (e.g., Gilovich et al., 2002). One form of deviation is notorious: People’s tendency to neglect base-rates in favor of specific case data. A number of theorists (e.g., Cosmides & Tooby, 1996; Brase, 2002a; Gigerenzer & Hoffrage, 1995) have argued that such neglect reveals little more than experimenters’ failure to ask about uncertainty in a form that naïve respondents can understand, specifically in the form of a question about natural frequencies. The brunt of our argument will be that this perspective is far too narrow. After surveying the theoretical perspectives on the issue, we will show that both data and conceptual considerations demand that judgment be understood in terms of dual processing systems, one that is responsible for systematic error and another that is capable of reasoning not just about natural frequencies, but about relations among any kind of set representation.

Base-rate neglect has been extensively studied in the context of Bayes’ theorem, which provides a normative standard for updating the probability of a hypothesis in light of new evidence. Research has evaluated the extent to which intuitive probability judgment conforms to the theorem by employing a Bayesian inference task in which the respondent is presented a word problem and has to infer the probability of a hypothesis (e.g., the presence versus absence of breast cancer) on the basis of an observation (e.g., a positive mammography). Consider the following Bayesian inference problem motivated by Eddy (1982; cf. Gigerenzer & Hoffrage, 1995):

The probability of breast cancer is 1% for a woman at age forty who participates in routine screening [base-rate]. If a woman has breast cancer, the probability is 80% that she will get a positive mammography [hit-rate]. If a woman does not have breast cancer, the probability is 9.6% that she will also get a positive mammography [false-alarm rate]. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer? __%

Early studies evaluating Bayesian inference under single-event probabilities also showed systematic deviations from Bayes’ theorem. Hammerton (1973), for example, found that only 10 per cent of the physicians tested generated the Bayesian solution, with the median response approximating the hit-rate of the test. Similarly, Casscells, Schoenberger, and Grayboys (1978) and Eddy (1982) found that a low proportion of respondents generated the Bayesian solution: 18 per cent in the former and 5 per cent in the latter, with the modal response in each study corresponding to the hit-rate of the test. All of this suggests that the mind does not normally reason in a way consistent with the laws of probability theory.

1.1. Base-rate facilitation

However, this conclusion has not been drawn universally. Eddy’s (1982) problem concerned a single event, the probability that a particular woman has breast cancer. In some problems, when probabilities that refer to the chances of a single event occurring (e.g., 1 %) are reformulated and presented in terms of natural frequency formats (e.g., 10 out of 1,000), people more often draw probability estimates that conform to Bayes theorem. Consider the following mammography problem presented in a natural frequency format by Gigerenzer and Hoffrage (1995).

10 out of every 1,000 women at age forty who participate in routine screening have breast cancer [base-rate]. 8 out of every 10 women with breast cancer will get a positive mammography [hit-rate]. 95 out of every 990 women without breast cancer will also get a positive mammography [false-alarm rate]. Here is a new representative sample of women at age forty who got a positive mammography in routine screening. How many of these women do you expect to actually have breast cancer? ___ out of ___.

The proportion of responses conforming to Bayes’ theorem increased by a factor of about three in this case, 46 per cent under natural frequency formats versus 16 per cent under a single-event probability format. The observed facilitation has motivated researchers to argue that coherent probability judgment depends on representing events in the form of natural frequencies (e.g., Cosmides & Tooby, 1996; Brase, 2002a; Gigerenzer & Hoffrage, 1995).

Cosmides and Tooby (1996) also conducted a series of experiments that employed Bayesian inference problems that had previously elicited judgmental errors under single-event probability formats. In Experiment 1, they replicated Casscells et al. (1978), demonstrating that only 12 per cent of their respondents produced the Bayesian answer when presented single-event probabilities. Cosmides and Tooby then transformed the single-event probabilities into natural frequencies, resulting in a remarkably high proportion of Bayesian responses: 72 per cent of respondents generated the Bayesian solution, supporting the author’s conclusion that Bayesian inference depends on the use of natural frequencies.

Gigerenzer (1996b) explored whether physicians, who frequently assess and diagnose medical illness, would demonstrate the same pattern of judgments as clinically untrained college undergraduates. Consistent with the judgments drawn by college students (e.g., Gigerenzer & Hoffrage, 1995), Gigerenzer found that the sample of 48 physicians tested generated the Bayesian solution in only 10 per cent of the cases under single-event probability formats whereas 46 per cent did with natural frequency formats. Physicians spent about 25 per cent more time on the single-event probability problems, suggesting that they found these problems more difficult to solve than problems presented in a natural frequency format. Thus, the physician’s judgments were consistent with those of non-physicians, suggesting that formal training in medical diagnosis does not lead to more accurate Bayesian reasoning and that natural frequencies facilitate probabilistic inference across populations.

Further studies have demonstrated that the facilitory effect of natural frequencies on Bayesian inference observed in the laboratory has the potential for improving the predictive accuracy of professionals in important real-world settings. Gigerenzer and his colleagues have shown, for example, that natural frequencies facilitate Bayesian inference in AIDS counseling (Gigerenzer et al., 1998), in the assessment of statistical information by judges (Lindsey et al., 2003), and in teaching Bayesian reasoning to college undergraduates (Sedlmeier & Gigerenzer, 2001; Kuzenhauser & Hoffrage, 2002). In summary, the reviewed findings demonstrate facilitation in Bayesian inference when single-event probabilities are translated into natural frequencies, consistent with the view that coherent probability judgment depends on natural frequency representations.

1.2. Theoretical accounts

Explanations of facilitation in Bayesian inference can be grouped into five types that can be arrayed along a continuum of cognitive control, from accounts that ascribe facilitation to processes that have little to do with strategic cognitive processing to those that appeal to general-purpose reasoning procedures. The five accounts we discuss can be contrasted at the coarsest level on five dimensions (see Table 1). We do not claim that theorists have consistently made these distinctions in the past, only that these distinctions are in fact appropriate ones.

Table 1. Prerequisites for reduction of base-rate neglect according to 5 theoretical frameworks.

	Mind as Swiss army knife	Natural frequency algorithm	Natural frequency heuristic	Non-evolutionary natural frequency heuristic	Nested sets and dual processes
Cognitive impenetrability	X
Informational encapsulation	X	X
Appeal to evolution	X	X	X
Cognitive process uniquely sensitive to natural frequency formats	X	X	X	X
Transparency of nested set relations	X	X	X	X	X

Note. The prerequisites of each theory are indicated by an ‘X’.

A parallel taxonomy for theories of categorization can be found in Sloman, Lombrozo, and Malt (in press). We briefly introduce the theoretical frameworks here. The discussion of each will be elaborated as required to reveal assumptions and derive predictions in the following sections in order to compare and contrast them.

1.2.1. Mind as Swiss army knife

Several theorists have argued that the human mind consists of a number of specialized modules (Cosmides & Tooby, 1995; Gigerenzer & Selten, 2001). Each module is assumed to be unavailable to conscious awareness or deliberate control (cognitively impenetrable) and able to process only a specific type of information (informationally encapsulated; see Fodor, 1983). One module in particular is designed to process natural frequencies. This module is thought to have evolved because natural frequency information is what was available to our ancestors in the environment of evolutionary adaptiveness. On this view, facilitation occurs because natural frequency data are processed by a computationally effective processing module.

Theorists who take this position uniformly motivate their hypothesis via a process of natural selection. However, the cognitive and evolutionary claims are in fact conceptually independent. The mind could consist of cognitively impenetrable and informationally encapsulated modules whether or not any or all of those modules evolved for the specific reasons offered.

1.2.2. Natural frequency algorithm

A weaker claim is that the mind includes a specific algorithm for effectively processing natural frequency information (Gigerenzer & Hoffrage, 1995). Unlike the mind-as-Swiss-army-knife view, this hypothesis makes no general claim about the architecture of mind. Despite this difference in scope, these theories adopt the same computational and evolutionary commitments.

Consistent with the mind-as-Swiss-army-knife view, this approach proposes that coherent probability judgment derives from a simplified form of Bayes’ theorem. The proposed algorithm computes the number of cases where the hypothesis and observation co-occur, N(H and D), out of the total number of cases where the observation occurs, N(H and D) + N(not-H and D) = N(D) (Kleiter, 1994; Gigerenzer & Hoffrage, 1995). Because this form of Bayes’ theorem expresses a simple ratio of frequencies, we refer to it as “the Ratio.”

Following the mind-as-Swiss-army knife view, proponents of this approach have ascribed the origin of the Bayesian ratio to evolution. Gigerenzer and Hoffrage (1995, p. 686), for example, state “The evolutionary argument that cognitive algorithms were designed for frequency information, acquired through natural sampling, has implications for the computations an organism needs to perform when making Bayesian inferences…. Bayesian algorithms are computationally simpler when information is encoded in a frequency format rather than a standard probability format.” As a consequence, this view predicts that “Performance on frequentist problems will satisfy some of the constraints that a calculus of probability specifies, such as Bayes’ rule. This would occur because some inductive reasoning mechanisms in our cognitive architecture embody aspects of a calculus of probability” (Cosmides & Tooby, 1996, p. 17).

The proposed algorithm is necessarily informationally encapsulated as it operates on a specific information format, natural frequencies, but it is not necessarily cognitively impenetrable as no one has claimed that other cognitive processes can’t affect or use the algorithm’s computations. The primary motivation for the existence of this algorithm has been computational (Kleiter, 1994; Gigerenzer & Hoffrage, 1995). As reviewed above, the value of natural frequencies is that these formats entail the sample and effect sizes and, as a consequence, simplify the calculation of Bayes’ theorem: Probability judgments are coherent with Bayesian prescriptions even without explicit consideration of base-rates.

1.2.3. Natural frequency heuristic

A claim that puts facilitation under more cognitive control is that people use heuristics to make judgments (Tversky & Kahneman, 1974; Gigerenzer & Selten, 2001) and that the Ratio is one such heuristic (Gigerenzer, Todd & the ABC research group, 1999). According to this view, “heuristics can perform as well, or better, than algorithms that involve complex computations…. The astonishingly high accuracy of these heuristics indicates their ecological rationality; fast and frugal heuristics exploit the statistical structure of the environment, and they are adapted to this structure” (Gigerenzer, 2006). Advocates of this approach motivate the proposed heuristic by pointing to the ecological validity of natural frequency formats, as Gigerenzer further states (p. 52), “To evaluate the performance of the human mind, one needs to look at its environment and, in particular, the external representation of the information. For most of the time during which the human mind evolved, information was encountered in the form of natural frequencies…” Thus, this view proposes that the mind evolved to process natural frequencies and that this evolutionary adaptation gave rise to the proposed heuristic that computes the Bayesian Ratio from natural frequencies.

1.2.4. Non-evolutionary natural frequency heuristic

Evolutionary arguments about the ecological validity of natural frequency representations provide part of the motivation for the preceding theories. In particular, proponents of the theories argue that throughout the course of human evolution natural frequencies were acquired via natural sampling (i.e., encoding event frequencies as they are encountered and storing them in the appropriate reference class).

In contrast, the non-evolutionary natural frequency theory proposes that natural sampling is not necessarily an evolved procedure for encoding statistical regularities in the environment, but a useful sampling method that, one way or another, people can appreciate and use. The natural frequency representations that result from natural sampling, on this view, simplify the calculation of Bayes’ theorem and, as a consequence, facilitate Bayesian inference (Kleiter, 1994). Thus, this view differs from the preceding accounts by resting on a purely computational argument that is independent of any commitments to which cognitive processes have been selected for by evolution.

This theory proposes that the computational simplicity afforded by natural frequencies gives rise to a heuristic that computes the Bayesian Ratio from natural frequencies. The proposed heuristic implies a higher degree of cognitive control than the preceding modular proposed algorithms.

1.2.5. Nested sets and dual processes

The most extreme departure from the modular view claims that facilitation is a product of general-purpose reasoning processes (Evans et al., 2000; Fox & Levav, 2004; Girotto & Gonzales, 2001; Johnson-Laird et al., 1999; Kahneman & Frederick, 2002, 2005; Over, 2003; Sloman et al., 2003). On this view, people use two systems to reason (Evans & Over, 1996; Kahneman & Frederick, 2002, 2005; Sloman, 1996; Stanovich & West, 2000), often called Systems 1 and 2 but in an effort to use more expressive labels, we will employ Sloman’s terms “associative” and “rule-based.”

The dual-process model attributes responses based on associative principles like similarity or retrieval from memory to a primitive associative judgment system. It attributes responses based on more deliberative processing that involves working memory such as the elementary set operations that respect the logic of set inclusion and facilitate Bayesian inference to a second rule-based system. Judgmental errors produced by cognitive heuristics are generated by associative processes, whereas the induction of a representation of category instances that makes nested set relations transparent also induces use of rules about elementary set operations, operations of the sort perhaps described by Fox and Levav (2004) or Johnson-Laird et al. (1999).

According to this theory, base-rate neglect results from associative responding and facilitation occurs when people correctly use rules to make the inference. Rule-based inference is more cognitively demanding than associative inference and is therefore more likely to occur when participants have more time, more incentives, or more external aids to make a judgment and are under fewer other demands at the moment of judgment. It is also more likely for people who have greater skill employing the relevant rules. This last prediction is supported by Stanovich and West (2000) who find correlations between intelligence and use of base-rates.

Rules are effective devices for solving a problem to the extent that the problem is represented in a way compatible with the rules. For example, long division is an effective method for solving division problems but only if numbers are represented using Arabic numerals; division with Roman numerals requires different rules. By analogy, the reason that natural frequencies facilitate use of base-rates on this view is that the rules that people have access to and are able to use to solve the specific kind of problem studied in the base-rate neglect literature are more compatible with natural frequency formats than single-event probability formats.

Specifically, people are adept at using rules consisting of simple elementary set operations. But these operations are only applicable when problems are represented in terms of sets, as opposed to single events. According to this view, facilitation in Bayesian inference occurs under natural frequencies because these formats are an effective cue to the representation of the set structure underlying a Bayesian inference problem. This is the nested sets hypothesis of Tversky & Kahneman (1983). On this view, natural frequency formats prompt the respondent to adopt an outside view by inducing a representation of category instances (e.g., 10 out of 1,000 women have breast cancer) that reveals the set structure of the problem and makes the nested set relations transparent for problem solving[3]. We refer to this hypothesis as the nested sets theory (Ayton & Wright, 1994; Evans et al., 2000; Fox & Levav, 2004; Girotto & Gonzalez, 2001, 2002; Johnson-Laird et al., 1999; Kahneman & Tversky, 1983; Macchi, 2000; Mellers & McGraw, 1999; Sloman et al., 2003). Unlike the other theories, it predicts that facilitation should be observable in a variety of different tasks, not just posterior probability problems, when nested set relations are made transparent.

2.0. Overview of empirical and conceptual issues reviewed

We now turn to an evaluation of these five theoretical frameworks. We evaluate a range of empirical and conceptual issues that bear on the validity of these frameworks.

2.0. Review of empirical literature

The theories are evaluated with respect to the empirical predictions summarized in Table 2. The predictions of each theory derive from (i) the degree of cognitive control attributed to probability judgment (see Table 1), and (ii) the proposed cognitive operations that underlie estimates of probability.

Theories that adopt a low degree of cognitive control — proposing cognitively impenetrable modules or informationally encapsulated algorithms — restrict Bayesian inference to contexts that satisfy the assumptions of the processing module or algorithm. In contrast, theories that adopt a high degree of cognitive control — appealing to a natural frequency heuristic or a domain general capacity to perform set operations — predict Bayesian inference in a wider range of contexts. The latter theories are distinguished from one another in terms of the cognitive operations they propose: The evolutionary and non-evolutionary natural frequency heuristics depend on structural features of the problem like question form and reference class. They imply the accurate encoding and comprehension of natural frequencies and an accurate weighting of the encoded event frequencies to calculate the Bayesian ratio. In contrast, the nested sets theory does not rely on natural frequencies and instead predicts facilitation in Bayesian inference, and in a range of other deductive and inductive reasoning tasks, when the set structure of the problem is made transparent, thereby promoting use of elementary set operations and inferences about the logical (i.e., extensional) properties they entail.

Table 2. Empirical predictions of the five theoretical frameworks.

	Mind as Swiss army knife	Natural frequency algorithm	Natural frequency heuristic	Non-evolutionary natural frequency heuristic	Nested sets and dual processes
Facilitation with natural frequencies (information format and judgment domain)	X	X	X	X	X
Facilitation with questions that prompt the respondent to compute the Bayesian ratio (question form)			X	X	X
Facilitation with statistical information organized in a partitive structure (reference class)			X	X	X
Facilitation with diagrammatic representations that highlight the set structure of the problem			X	X	X
Inaccurate frequency judgments					X
Equivalent comprehension of natural frequencies and single-event probabilities					X
Non-normative weighting of likelihood ratio and prior odds					X
Facilitation with set representations in deductive and inductive reasoning					X

Note. The predictions of each theory are indicated by an ‘X.’

2.1. Information format and judgment domain

The preceding review of the literature found that natural frequencies formats consistently reduced base-rate neglect relative to probability formats. However, the size of this effect varied considerably across studies (see Table 3).

Cosmides and Tooby (1996), for example, observed a 60-point percent difference between the proportions of Bayesian responses under natural frequencies versus single-event probabilities, whereas Gigerenzer and Hoffrage (1995) reported a difference only half that size. The wide variability in the size of the effects makes it clear that in no sense do natural frequencies eliminate base-rate neglect, though they do reduce it.

Sloman, Over, Slovak, and Stibel (2003) conducted a series of experiments that attempted to replicate the effect sizes observed by the previous studies (e.g., Cosmides & Tooby, 1996, Experiment 2, Condition 1). Although Sloman et al. found facilitation with natural frequencies, the size of the effect was smaller than that observed by Cosmides and Tooby: The percent of Bayesian solutions generated under single-event probabilities (20%) was comparable to Cosmides and Tooby (12%), but the percentage of Bayesian answers generated under natural frequencies was smaller (i.e., 72% versus 51% for Sloman et al.). In a further replication, Sloman et al. found that only 31 per cent of their respondents generated the Bayesian solution, a statistically non-significant advantage for natural frequencies.

Table 3. Percent correct for Bayesian inference problems reported in the literature (sample sizes in parentheses)

Information format and judgment domain

Study Probability Frequency

Casscells et al., (1978) 18 (60) ---

Cosmides & Tooby (1996; Exp. 2) 12 (25) 72 (25)

Eddy (1988) 5 (100) ---

Evans et al., (2000; Exp. 1) 24 (42) 35 (43)‡

Gigerenzer (1996) 10 (48) 46 (48)

Gigerenzer & Hoffrage (1995) 16 (30) 46 (30)

Macchi (2000) 6 (30) 40 (30)

Sloman et al., (2003) (Exp.1) 20 (25) 51 (45)

Sloman et al., (2003) (Exp. 1b) --- 31 (48)‡

Note. Probability problems require that the respondent compute a conditional-event probability from data presented in a non-partitive form, whereas frequency problems include questions that prompt the respondent to evaluate the two terms of the Bayesian ratio and present data that is partitioned into these components.

‡ p > 0.05

Evans, Handley, Perham, Over, and Thompson, (2000; Experiment 1) similarly found only a small effect of information format. They report 24 per cent Bayesian solutions under single-event probabilities and 35 per cent under natural frequencies, a difference that was not reliable.

Brase, Fiddick, and Harries (in press) examined whether methodological factors contribute to the observed variability in effect size. They identified two factors that modulate the facilitory effect of natural frequencies in Bayesian inference: (1) the academic selectivity of the university the participants attend, and (2) whether or not the experiment offered a monetary incentive for participation. Experiments whose participants attended a top-tier national university and were paid reported a significantly higher proportion of Bayesian responses (e.g., Cosmides & Tooby, 1996) than experiments whose participants attended a second-tier regional university and were not paid (e.g., Brase et al., in press, Experiments 3 and 4). These results suggest that a higher proportion of Bayesian responses is observed in experiments that (a) select participants with a higher level of general intelligence, as indexed by the academic selectivity of the university the participant attends (Stanovich & West, 1998), and (b) increase motivation by providing a monetary incentive. The former observation is consistent with the view that Bayesian inference depends on domain general cognitive processes to the degree that intelligence is domain general. The latter suggests that Bayesian inference is strategic, and not supported by automatic (e.g., modularized) reasoning processes.

2.2. Question form

One methodological factor that may mediate the effect of problem format is the form of the Bayesian inference question presented to participants (Girotto & Gonzalez, 2001). The Bayesian solution expresses the ratio between the size of the subset of cases where the hypothesis and observation co-occur and the total number of observations. Thus, it follows that the respondent should be more likely to arrive at this solution when prompted to adopt an outside view by utilizing the sample of category instances presented in the problem (e.g., “Here is a new sample of patients who have obtained a positive test result in routine screening. How many of these patients do you expect to actually have the disease? ___ out of ___”) versus a question that presents information about category properties (e.g., “… Pierre has a positive reaction to the test…”) and prompts the respondent to adopt an inside view by considering the fact about Pierre to compute a probability estimate. As a result, the form of the question should modulate the observed facilitation.

In the preceding studies, however, information format and judgment domain were confounded with question form: Only problems that presented natural frequencies prompted use of the sample of category instances presented in the problem to compute the two terms of the Bayesian solution (an outside view), whereas single-event probability problems prompted the use of category properties to compute a conditional probability.

To dissociate these factors, Girotto and Gonzalez (2001) proposed that single-event probabilities (e.g., 1%) can be represented as chances[4] (e.g., “1 chance out of 100”). Under the chance formulation of probability, the respondent can be asked either for the standard conditional probability or for values that correspond more closely to the ratio expressed by Bayes’ theorem. The latter question asks the respondent to evaluate the chances that Pierre has a positive test and the infection, out of the total chances that Pierre has a positive test, thereby prompting consideration of the chances that Pierre — who could be anyone with a positive test in the sample — has the infection. In addition to encouraging an outside view, this question prompts the computation of the Bayesian ratio in two clearly defined steps: First calculate the overall number of chances where the conditioning event is observed, then compare this quantity to the number of chances where the conditioning event is observed in the presence of the hypothesis.