A remarkable thing happened in the mid 1980's. For the first time we could actually look at pictures of the human brain while people thought. The pictures were in terms of areas of increased blood flow caused by enhanced neural activity during mental effort. The studies showed relatively local areas of increased activity that we found could be related to the specific computations being carried out by the person.
In the ensuing decade many others have examined such pictures and new methods and theories concerning their meaning have developed. It has become rather common place to show detailed pictures of areas of active blood flow in a variety of tasks. These pictures have been presented in newspapers, popular magazines, major scientific journals and on TV. Several major world centers have developed programs of research to image changes in aspects of the blood supply using PET or functional MRI.
Nothing can quite describe our excitement during our first scans in seeing how specific the brain's activity was and how beautifully it adapted to the requirements of the task. We describe the origin of our work in the preface.
This book is about what has been learned from this effort to picture the mental processes of people as they perceive and think. It originated in a scientific collaboration that began in 1985 when Michael Posner went to St. Louis to work with Marcus Raichle. Raichle had created a leading center for studies using positron emission tomography.
Both of us wanted to see if PET could be used as a tool to picture mental processes in human beings. In our book we try to provide a general motivation and framework for relating these pictures to theories of cognitive function. The book itself has 165 illustrations as well as a great deal of narrative.
The first chapter provides a historical background and the second chapter deals with the effort of cognitive psychologists to measure mental operations. The third chapter introduces the PET method, gives evidence on its validation and applies it to passive reception of visual information. The fourth chapter deals with active processes of visual attention, imagery and search. In chapter 5 we use visual words to illustrate how brain imaging can reveal the organization of high level learned skills. Chapter 6 involves our effort to go from functional anatomy to circuitry by use of high density electrical recording to obtain the time course of brain activations. In Chapter 7 we review studies of the brain networks involved in attention. We argue that separate brain systems are involved in the control of information flow in the human brain. Chapter 8 deals with the development of attentional networks in early infancy. Chapter 9 applies the brain imaging work to the study of pathologies. Chapter 10 provides our speculations about the future of imaging studies. Below we provide very brief material from each of these chapters to give the reader an overview of our efforts.
PAST IMAGES
We wanted to place the PET results in an appropriate historical context. In our first chapter we deal with past images of mind brain interaction. This historical context includes more than a century of efforts to relate cognitive processes to the brain. We describe how the study of reflexes lead to a very sophisticated idea of cell assemblies developed by D.O. Hebb.
It was 43 years from the appearance of Sechenov's classic work until Sherrington's integration. Exactly 43 years later, in 1949, the Canadian psychologist Donald O. Hebb produced an immensely influential book, The Organization of Behavior: A Neuropsychological Theory, that provided an exciting integration of mental and brain processes. Hebb took up the challenge of integrating psychology and physiology by providing the first testable theory of how neural circuits might support mental processes such as attention and memory.
Hebb rejected the assumption that behavior was only a series of responses to environmental stimulation (a position endorsed by Sechenov and not rejected by Sherrington). At the heart of his theorizing was the concept of "cell assemblies," a powerful idea that remains a major conceptual force in theories of brain function. Hebb thought that any frequently repeated stimulation would lead to the development of a structure consisting of neurons capable of acting together as a closed system. This structure was diffuse: the neurons of a cell assembly were probably not located in a single place but spread throughout the brain.
THE STUDY OF COGNITION
At the turn of the century students of what was termed phrenology attempted to correlate prominent landmarks of the skull and even the face with psychological processes. They proposed specific areas for complex tasks such as processing language, forming mental images, or reasoning and for individual traits such as authority, wisdom, and religiosity. Phrenology has often been criticized as a pseudoscience because there was absolutely no scientific reason to believe that bumps on the skull were related in any way to the brain tissue that underlay them. But its critics usually overlooked that phrenology was based on an analysis, admittedly crude, of the thought processes and behaviors of our daily lives. For the phrenologists the complex activities of our daily lives were the products of faculties, or constituents of the mind, that affected brain activity and could be observed in external features of the skull.
The idea of breaking down complex activities into their constituent parts was advanced substantially in the late 1950s through the efforts to develop artificial intelligence. The General Problem Solver (GPS), for example, was a computer program developed primarily by Alan Newell and Herbert Simon at Carnegie Mellon University capable of proving theorems that had been shown previously to form the basis of mathematics. GPS was able to prove theorems by breaking the problem into small steps and searching systematically within the space of possible solutions. The ability to program a computer to perform tasks that seemed to involve the very highest levels of human reasoning had a dramatic effect upon psychology. One important consequence was to stimulate efforts to analyze in detail the constituent operations involved in human thought.
Hebb's work set the occasion for cellular studies in anaesthetized and alert animals that have done so much to influence current ideas of the cellular basis of perception. The idea of relating cerebral blood flow to cognition is also an old one, but the development of convenient methods to achieve this has taken place only in the last ten years.
BRAIN IMAGING TECHNIQUES
After the introduction of x-ray CT, scientists familiar with autoradiography began to consider the possibility of performing similar autoradiographic measurements in humans. The major advantage of the method envisioned was that it would render the physical removal of the organ of interest unnecessary. Instead, an image of the organ would be reconstructed "electronically". In other words, scientists suddenly realized that if they could reconstruct the density and, hence, the anatomy of an organ by passing x-rays through it, they could also reconstruct the distribution of a previously administered radioisotope in any desired section of the body by recording its emissions. With this realization, around the year 1973, was born the whole idea of autoradiography of living humans subjects. Some scientists and engineers immediately set about constructing instruments that could detect the emissions, while others considered what form of radioactive emission they would measure.
MEASURING MIND
Chapter 2 of our book is an effort to try to develop an understanding of the measurement of mental operation by chronometric means. In 1978 one of us devoted a whole book to this topic, but in this chapter we briefly describe how scientists study mental operations in objective terms. Without the ability to quantify mental activity they would be lost when trying to understand what the brain is doing. It is often thought that mental events cannot be directly measured because they are private and immaterial. However, psychologists have devised ways to measure these events objectively by taking advantage of the fact that all mental operations take time. Even a seemingly private or personal experience like creating a mental image of one's home or favorite pet can be examined objectively by observing its influence on the speed of response to probe events. Just as the trace of an invisible particle can be seen by the physicist by its effects on other particles, a mental operation becomes visible when it influences the time taken to respond to a specific probe question presented by the experimenter.
In this chapter we also provide a framework for linking mental operations to underlying brain systems. We view brain imaging methods as a crucial way of testing whether mental operations can be localized in particular neural system and as a start toward specifying mental operations in neural terms.
IMAGES OF THE BRAIN
One way to check the validity of the PET technique is to compare PET scans with the known facts of how neuronal activity changes.
The best understood of the brain's cognitive activities is the passive processing of visual stimuli--the experiences of visual objects that flood in when we open our eyes, even when we make no effort to actively process the visual input. Our knowledge of the brain pathways activated during passive perception makes a good standard against which we can compare PET images. If the match is good, we have evidence that supports the use of PET. Because passive processing uses the simplest visual pathway, with the fewest brain areas activated, it is the base on which we can build our understanding of more active cognitive processes, such as thought.
In order to test the ability of PET to map the retinotopic organization of human striate cortex, we chose as our stimulus an annular checkerboard, which was presented on a television monitor and controlled by a computer. The divisions within the checkerboard (alternately red and black) switched colors at the rate of 10 times per second. We had previously determined that this frequency produced the optimal blood flow response in the primary visual cortex. Under the control of the computer we were able to vary the size of the checkerboard and the fraction seen by our subjects - full annulus, upper half, lower half, right half, or left half.
During the control condition for this studies, subjects focused on a tiny white dot in the middle of the television monitor while we measured blood flow. In every subject blood flow was measured alternately in the control state and during various presentations of the checkerboard. When viewing the checkerboard, the subjects were also asked to fix their gaze on the same tiny white dot, which was always clearly visible in the center of the checkerboard. Collecting that data in this manner fulfilled our requirements for paired image subtractions and image averaging.
The results of these studies were gratifying. In this first group of subjects a full annular checkerboard was presented either to the center of the fovea, to an area around the fovea but not including it, or to the periphery of the retina. Entirely as predicted, the averaged PET images of blood flow revealed the known retinotopic organization of the primary visual cortex. As the annular checkerboard stimulated ever more peripheral portions of the retina, the location of the blood flow response moved systematically forward in the brain.
With a spatial resolution of only 18 millimeters you might legitimately wonder how we are able to confidently accept the results of our retinotopy experiments. For example when we recorded the blood flow responses to fovea, near-fovea, and peripheral stimulation, the distances between he three centers of activation were certainly much less than 18 millimeters. The key to our success is not some form of trickery or unscientific handwaving. Rather, it is our ability to estimate with an accuracy of 1 to 2 millimeters the central tendencies in these foci of blood flow activation. The estimate is made by computer suing what is descriptively known as a "center-of-mass" algorithm. The principles upon which the algorithm is based are well grounded in signal detection theory. Interestingly, the very visual system we are studying in these experiments employs somewhat the same strategy. Our visual performance is much better than predicted by the inherent spatial solution of our retina. This enhanced performance, referred to as "hyperacuity," results from many of the same principles we employ in PET to detect the location of blood flow responses.
VISUAL WORDS
How are complex learned stimuli registered in the visual system?
We tried to answer that question in a PET study. The study required subjects to fix their gazes on a point in the middle of a television monitor. Common English nouns appeared below the fixation point, one after another, while we performed the PET scan. The instructions were simply to look at the fixation point. From this scan we subtracted a second scan taken while the subjects looked at the fixation point but no words were presented. We already knew the response of the brain to passively viewing the fixation point, so this made a nice control scan for our study.
The results revealed a much more extensive activation of the visual system from viewing words than was encountered by simply opening the eyes and viewing fixation point. Areas of activations were found both to the left and right in the extrastriate cortex. What were we to make of these additional areas of activation? Further analysis of words as visual stimuli was clearly needed.
In studying words as a visual stimuli, we considered at least four codes. First, words must be viewed as complex collections of visual features. They consist of varying numbers of units made up of connected lines with varying spatial orientations. Second, these units of connected lines are not a random arrangement of lines, but come from a unique set of 26 units representing the letters of the English alphabet. Third, the letters in words are not randomly assigned, but are arranged in combinations of vowels and consonants according to the rules of the English language. These rules reflect the ways in which English letters can be put together to make words (orthography) and, to some extent, the ways in which the letters are pronounced. Finally words convey meaning.
It is reasonable to suppose that the response we observed in the human visual system to the passive presentation of words represents a combination of responses to any or all of these features of words. We wished to connect each activated area to a specific feature. To dissect the overall response into its parts, we needed to determine the component operations involved in reading a word. We believed at the onset that these operations related to visual features, letters, and letter combinations, but not to the word meaning, which we did not expect to be processed in the visual system. We devised four different visual stimuli, shown in the table on the next page, that systematically varied the different features of words. We then observed the effect of these different stimuli on the visual system of the brain as recorded by a PET scan. The four stimuli consisted of false fonts, consonant letter strings, pseudowords, and words; each represented one of the four suggested codes. The computer-generated false fonts incorporated all of the visual features of words but none of the features from any other codes. Consonant letter strings added the letter code but obviously could not be pronounced according to English rules of pronunciation. Pseudowords contained both consonants and vowels and could be pronounced because they had been arranged according to the pronunciation rules of English. Four most individuals, pseudowords do not convey meaning (an occasional individual will recognize an Hungarian word or the name of a neighbor inadvertently included among the pseudowords). Finally, words themselves contain all the aforementioned features plus meaning.
A group of normal individuals were instructed to observe the stimuli passively. By scanning the subjects as they were shown false fonts, consonant letter strings, pseudowords, or words, it was possible to distinguish among responses to the visual features, letter, orthographic regularity, and meaning. If the activation to words was due to the visual features of the stimulus and nothing else, then all stimuli should produce the same response. If activation was due to the letter code, then the activation should be produced only by letter strings, pseudowords, and words. If the activation was due to the orthographic regularity of words, regardless of their meaning, then the results produced with words and pseudowords should be identical. And, finally, if the activation to words was due to the meaning of words, even during passive presentation, then words would produce a unique response.
Thus, using the methods of cognitive science, we have broken down the passive perception of words into a group of component operations. We can now ask the question, how are these component operations implemented in the brain? All four groups of stimuli produced response in visual areas outside of the primary visual cortex, as these areas reacted to the complex visual features of the stimuli regardless of whether they consisted of groups of false fonts or letters. However, only pseudowords and words produced the dramatic responses originally observed with words alone. These unique responses occurred in areas outside of the primary visual cortex, along the inner surface of the left cerebral hemisphere.
Two levels of analysis appear to be occurring in the visual system as we passively view words. At one level, the brain analyzes the visual features of the stimuli regardless of their relationships to letters and words. These visual features appear to be processed in multiple areas of the visual system on both sides of the brain. Responses to the false font that contains only meaningless features are particularly strong in the right hemisphere. At the second level, the brain analyzes the visual word form. It seems clear that visual stimuli incorporating the orthographic rules of the English language uniquely activate a group of areas in the visual system of fluent readers of English. This coordinated response among a group of areas clustered in one part of the visual system must be acquired as we learn to read, and its existence is probably critical to the facility with which skilled readers handle words. As our knowledge of the organization of the visual system grows, we anticipate that the identity of the constituent members of this specialized grouping of extrastriate visual areas will be revealed and their individual contributions to the process explained.
The results reviewed here already provide important support for the use of PET studies. First, these studies allow us to locate areas of the brain associated with specific mental operations. The results seem reliable from study to study when we can be sue that the same function is involved, as in the studies of color and motion. Moreover, the PET methods can be extended to deal with complex stimuli such as words, and they can reveal unique areas of the brain associated with functions such as reading that we have learned to carry out on these objects. In the next chapter we extend our PET studies from those in which subjects are passive to those in which they must attend actively to the stimuli.
VISUAL ATTENTION
The PET studies discussed in Chapter 3 indicated those areas of the human brain showing increased blood flow during the passive reception of color, motion, and form. We now consider how these and other brain areas are affected when subjects are asked to attend to these same visual attributes, in order to select a target. In one study conducted to address this issue, two visual stimuli were presented one after the other, separated by a brief interval. Each stimulus consisted of up to 30 identical visual elements in a frame. In a single trial, two frames were presented for 400 milliseconds each, with a 200-millisecond pause in between. In half the trials, the two frames were identical. In the other half, all of the elements changed in color, shape, or velocity, or in some combination of these attributes.
Subjects were given one of five different instructions to follow, representing five different conditions of the experiment. In the passive control condition, the subject was merely to observe the display and make no decision. In the "divided attention" condition, subjects were asked to press a key if any o the elements changed. In three 'focal attention" conditions, subjects were to react only when a specified attribute changed: they were required to report either color changes only, form changes only, or motion changes only. Our goal was, first, to find out if attending to a particular attribute, as compared to attending to a particular attribute, as compared to attending to all attributes, activated the same brain areas that had previously been found to be activated under passive instructions. Second, people perform better when they attend, and we wondered whether these brain areas were more strongly activated during attention than during passive perception.
We found that subjects detected targets more accurately in each of the three focal conditions than in the divided condition. The percentage of correct detections was higher, and the percentage of false detections when no target was presented was lower. Evidently, attending to a single type of target facilitates performance compared to attending to all types of targets. We had expected these results and indeed had depended on them in order to be able to analyze the data. We planned to subtract the divided attention PET image from each of the three focal attention PET images, so that we could obtain a picture of the brain areas involved when subjects restrict attention to a given type of sensory information such as motion. But, if subjects could attend to all the attributes simultaneously with the same efficiency as to any one, the divided condition would show exactly the same blood flow as each of the focal conditions, and image subtractions would be useless.
Because divided attention is less efficient as expected, we were in fact able to subtract the divided attention PET image from each focal attention PET image. The stimuli presented and the number of key presses were identical in the focal and divided attention conditions; only what the person was to report as a target differed. This, we could be confident that the subtraction was isolating those areas, and only those areas, activated when the subject attended to a particular attribute such as motion. If our views were correct, we should find more activity in those attended areas in the subtracted image. The logic of this subtraction is thus to determine if attention to motion, for example, amplifies neuronal activity in the area of the brain related to motion and thus increases blood flow in that area.
This logic rests on a number of assumptions. First, there must be areas of the brain specialized to perform computations based on color, motion, or form. Considerable evidence exists for such specialized visual areas in the monkey brain, and Chapter 3 describes evidence for these areas in the human brain from PET studies of passive perception. Second, the instruction to attend to motion, color, or form must induce subjects to select the information about the relevant attribute. If it does, we should find that subjects perform better when they know what attribute they should attend to than when they do not. This assumption was borne out by the results. Third, attending to a particular attribute should facilitate computation in the area of the brain specialized to process that attribute, and this facilitation should be measurable by increases in blood flow.
This is a complex logic, and there could be several reasons why such an experiment might not work. However, it was clear that each focal condition activates a unique set of visual areas in the extrastriate cortex. Moreover, there is a strong correspondence between the areas activated during this experiment and the areas activated during passive stimulation by color and motion.
IMAGINING WORDS: THE COGNITIVE STUDIES
In studying how visual imagery is constructed, we gather evidence from three main sources: cognitive studies of normal subjects, studies of brain-damaged patients, and most recently, neuroimaging studies. As usual, all these studies require the subjects to perform tasks, but in designing the tasks the idea is to eliminate all the stages of processing activated automatically by a visual stimulus. That way, characteristics of the visual experience can be studied in isolation from sensory events. In this section, we argue that the three methods converge in suggesting that when we create visual images the same extrastriate areas are involved that are activated passively by visual stimuli. In addition, we again find that the activity of these areas is amplified by the act of attention required to process a specific target.
We will show that when we construct an image, a number of the mental operations resemble those that occur when the stimulus is physically presented. In other words, there are fundamental similarities between perceiving an image and imagining one. However, some informal evidence might seem to suggest that something very different happens when we create mental image. One such piece of evidence is the difficulty we all have in recalling word from memory and spelling it backward. Form a visual image of the word "pumpkin", for example. Can you spell it backward? You probably cannot do the task efficiently. Our general inability to do so has sometimes led psychologists to suppose that images in general are linear strings of symbolic items like the names of the letters in a word, rather than true visual images. They reason that when seeing a word one can easily scan it in either direction. As a result, spelling a word in clear view backward is relatively easy. Thus, if our memory of the word was in the firm of a visual image, we should be able to scan it in either direction with nearly equal ease. However, if the memory is in the form of a series of symbols ordering from first letter to last, we would be likely to have the same trouble spelling the word backward as we do reciting the alphabet. To compare in detail the mental operations underlying perception and image formation, it is important to have more than subjective impressions or anecdotes to go on. One objective test was invented by Robert Weber, then at Oklahoma State University. He asked subjects whether each letter of a string was contained within the central line (e.g., a, c) or had an ascender (d, l) or a descender (q, g). To answer this question in the absence of the printed word, most people attempt to form images of the letters. Weber compared the time needed to search the letter string when it is physically present on the retina with the time needed when it is present as an image summoned from memory. He found that when the string contains three letters, subjects are just as fast and accurate in scanning a mental image as in scanning a physically present letter string, regardless of whether it is scanned forward or backward. Beyond three letters in length, the memory image becomes quite inefficient, and subjects are far better off if the string is present in front of them. By this objective test, the image is like a percept for three-letter words, but not for longer ones. The results suggest that only a few items can be imaged at one time. Thus, imagery and perception behave differently for long words like "pumpkin," but not for short words like "pin." Weber's experiment suggests that the processing of imagery and the processing of percepts have in common some of their detailed mental operations. According to general framework outlined in Chapter 2, it would be useful to ask if these common operations arise as functions of the same brain areas. To explore this issue, scientists have turned to studies of people with brain damage. Do patients with similar types of brain damage have similar problems with perceiving and imagining? If so, then the same brain areas may well be activated in both kinds of tasks.
Many patients with damage to a parietal lobe show "neglect" on the side of visual objects opposite their brain-damaged hemisphere; these same patients have similar problems in scanning visual images constructed in their minds. Eduardo Bisiach, an Italian neurologist, studied patients with damage to the right parietal lobe. He first asked them to imagine a familiar square in their native Milan as it looks while standing in the square facing the church. They reported most objects on their right but very few on their left. When asked to image the scene from the opposite perspective, while standing on the church stairs looking out in the square, they neglected objects previously reported and reported objects previously neglected. They showed the same kind of neglect, on the same side, as they reported when looking at objects that were physically present in the field in vision.
NEUROIMAGING STUDIES
With our colleagues Julie Fiez and Steve Petersen at Washington University, we showed normal subjects a word in uppercase font and asked them to imagine whether the lowercase version of the word contained an ascender. Recall that an ascender is a letter like "d" or "l" that rises above the central line of a word. Subjects saw the uppercase cue word for 150 milliseconds and then has 1850 milliseconds before the next cue appeared to recall from memory and appearance of the word in lowercase letters, analyze it for the presence of an ascender, and report the results with a key press. We compared this task with one in which the subjects listened to a spoken word and decided whether it contained an ascender. The visual imagery task produced dramatic activation of the parietal as well as the temporal lobes on both sides, as shown on the left of the image on this page. Presenting the cue word to the ear rather than the eye greatly diminished the activity in the temporal lobes but had no effect on the activity in the parietal lobes. Thus, the temporal lobes may be concerned specifically with transforming the uppercase visual cue into its imagined lowercase form. Interestingly, the parietal lobe responses are identical with those obtained when subjects must attend to the spatial relationships of externally presented stimuli.
INTERPRETING WORDS
We asked subjects to perform the task of repeating a visually presented noun and a task of generating the use of each noun. We worked with lists of 40 common English nouns presented at the rate of 40 words per minute. We concluded that during practice the subjects didn't learn to say the words per se. Rather they learned to select a specific word in response to a specific stimulus. Having established a better appreciation of the effect of practice on a subject's performance in the generation task, we used this information to design a new experiment.
We tested the hypothesis that there are two pathways for this type of verbal response, one that generates learned, automatic responses and one that generates fresh, unlearned responses. A new group of 12 subjects were imaged while they performed the generation task, but this time under three different experimental conditions: (1) First they were given a list of nouns without any practice; (2) then they were given the same list of nouns as before but were scanned only after 15 minutes of intense practice on that list; and (3) after performing conditions 1 and 2, they were given a completely different list of nouns, with no practice. Subjects performing condition 3 were experienced in the task but not in the stimuli, whereas subjects performing condition 2 were experience din both and thus able to perform in a highly automated and effortless manner. The control task was the same as in the original experiment, simply reading aloud the nouns as they were presented on the television monitor. The results of this experiment, shown on this page, demonstrated dramatically the capacity of the brain, as the result of practice, to change the areas being used.
We believe the results are consistent with the hypothesis that two pathways exist for the execution of the generation task. One pathway includes the anterior cingulate cortex, the left frontal cortex (including Broca's area), the left temporal cortex (including Wernicke's area), and the right cerebellum, and the other includes the buried insular cortex in both hemispheres. In addition, practice enhances the response of the areas near the midline of the left visual cortex that we had earlier identified as uniquely responsive to orthographically regular symbols like words and pseudowords.
It is of course appropriate to ask whether practice had any effect on the control task that we used in these experiments, reading nouns aloud. The answer is no. This confirms what might seem obvious, that reading aloud common English nouns is an automatic task for skilled readers.
It is an old idea, first expressed in 1885 by the neurologist Ludwig Lichtheim, that different pathways exist in the brain for verbal responses that are generated automatically in response to sensory input and for those that require detailed analysis. Following the analysis of William James at the turn of the century, it has also been common to psychology to distinguish between automatic and controlled processes in many domains, including language. What PET contributes is an exquisite picture of how the brain adapts as new thoughts become automatic through practice. Generating a novel use of noun produces activation in frontal areas and reduces the activation of the more automatic routes used in word reading. As the same list of nouns is repeated, the additional activations drop away and the brain generates the response just as though subjects were performing the highly automatic task of reading the words aloud.
Generating the use of a noun repeatedly appears to be like reading words aloud under conditions that minimize semantic processing.
In this chapter we have sought to bring together studies of lexical access that have used a variety of experimental techniques. These have included cognitive methods, studies of patients with brain lesions, and functional brain imaging with PET. As information accumulates from all sources and, in particular, from the functional brain imaging of normal subjects, a clearer picture is emerging of how this uniquely human skill is implemented in the brain.
Results from the PET imaging studies bear out the importance of the cognitive models of lexical access with their multiple codes and parallel routes. Such models show real promise of explaining the neurological observations.
SEQUENCING MENTAL OPERATIONS
Despite the power of PET imaging to reveal the functional anatomy of the human brain, we do not gain from this tool an appreciation of how activity within circuits change over time. For example, how does information flow in the areas of the brain concerned with the generation task? Does activity in the frontal cortex precede that in the temporal cortex and cerebellum? When is the anterior cingulate cortex active? Questions such as these suggest that we need to consider the temporal dimension of brain organization. To accomplish this task we turned to measurements of electrical activity recorded from the scalp.
The first step was to establish that electrical recordings can indeed measure the activity coming from areas shown to be active during the PET studies. These studies had revealed two major areas of activation within the visual system. A visual area in the right hemisphere is activated by all types of character strings, and that activity appears to be enhanced when subjects are required to detect a particular feature. A visual area in the left hemisphere near the midline is activated by both words and nonwords that have lawful spellings in English, but not by consonant strings.
We examine the event-related potentials recorded from the scalp while subjects looked at words and consonant strings, and found two features that appeared to be related to the areas just described. First, we found a very strong asymmetry about 100 milliseconds after the stimulus was presented: recordings from sites in the posterior right hemisphere were larger than those on the left. The waveforms on this page, recorded from electrodes over the parietal lobes of the right and left hemispheres, clearly show a more active right hemisphere. The difference between hemispheres is quite strong even when the data are averaged over many subjects, although within a particular subject the asymmetry may be limited to one or two electrodes. This effect is consistent with the existence of an area in the right posterior lobe activated by visual features. It suggests that the differences found in the PET data are created about 0.1 second after the letter string is presented to the viewer.
The second feature of the event-related potential may be produced by the visual word form system, which responds only to words and pseudowords. We found evidence for a difference in the response to words and consonant strings that started about 150 milliseconds after the word appeared and was strongest over posterior sites in the left hemisphere. These findings are generally consistent with the PET studies, which show the visual word form area to be located in the anterior left occipital lobe near the midline. The source of the electrical activity can be only loosely determined from the scalp voltage data, however. Moreover, the difference between words and consonant strings starting at about 150 milliseconds is not very strong, although it is consistent over several studies.
If our ERP effect is coming from the visual word form area, as we believe our findings suggest, then this area makes the initial discrimination between word sand nonwords starting about 150 to 200 milliseconds after the word is seen. Since the PET result is an average over 40 seconds of activity, for all we knew the posterior locations could have been activated by feedback from a computation executed at a more anterior site. The event-related potentials rule out this possibility: they show that posterior electrodes give the earliest difference between words and consonant strings, thus suggesting that the distinction between words and consonants arises at posterior sites and is not fed back to those sites from other areas.
To study novel thoughts, we have developed a method of analysis that we call conjunctions. This method asks subjects to analyze visual and semantic features in the same task. For example, subjects may identify words that have a thick letter and represent a manufactured object. ERP recordings trace when the various anatomical areas perform their computations.
In our study we present subjects with single printed words. Half of the words have a thickened letter and half appear in normal font. Half refer to manufactured items and half to natural items. We ask subjects to respond yes when a word has both a tick letter and is manufactured. That is, subjects say yes only to words that have both the physical and semantic attributes and no to all other words. On one day subjects perform the thick letter task and are then asked to respond with the yes key if the stimulus has a thick letter and is manufactured item. We ask the subjects to first look for the visual feature and then make the semantic analysis. On another day we have subjects perform the semantic task and then ask them to respond with the yes key if the word is a manufactured item and has a thick letter. This time the priority is on the semantics and not the physical feature. However, in both tasks the conjunction to be computed is exactly the same. The inputs are identical and the responses (if correct) are identical, but the priorities given to the underlying computations are reversed.
We did not expect the subjects to actually compute the functions in a serial fashion. Our hope was that they would emphasize the computation assigned priority and perhaps complete it somewhat earlier than the other computation. If the subjects were trained on the thick letter task and then asked to respond to targets that had a thick letter and were manufactured items, we expected them to give priority to the visual search for a thick letter. To see if this is what happened, we look at the reaction times from this task. Consider the results from targets lacking one of the searched-for features. On these trials subjects can respond as soon as they verify that a word is missing one of the features: either it lacks a thickened segment or is in the wrong semantic category. In general the thick letter is processed somewhat faster so that subjects quit sooner when there is no thick letter present than when there is but the target is in the wrong semantic category. However, they are faster still if the search for a thick letter has been given priority by training. On the other hand, if a thick letter is present, they take much longer to reject the word because it lies in the wrong semantic category, and they are now faster if the semantic task has been given priority by training. Thus the reaction times suggest that we have been successful in getting the subjects to assign some priority in accordance with our instructions.
We can now consider images of the underlying brain activity. First, it is clear that under the two task orderings the waveforms differ at about 300 milliseconds following the presentation of a word. Event-related potentials from the left frontal area show that emphasizing the visual task delays semantic processing: there is more positive electrical activity after 300 milliseconds and a slower return to baseline. Event-related potentials from the right posterior region show the reverse: when the visual task is given priority, electrical activity is reduced after 300 milliseconds and the waveform returns to baseline earlier.
The conjunction method provides a very general tool for exploring the circuitry involved in brain activity. In general one thinks of brain circuitry as formed of fixed anatomical connections between brain areas or between neurons within an area. However, it is well known that any brain area can be anatomically connected to any other area by either direct or indirect routes. In higher cognition, the act of attending organizes the circuitry between brain areas. It is in this sense that attention can control the order of computations such as those involved in the thick letter and semantic category tasks. If this hypothesis concerning the brain is correct, conjunctions can be used to explore the relation between remote areas of the brain during specific cognitive tasks. The method uses the person's own attention to illuminate the order of the computations and thus the higher-level circuits that execute the task instructions.
In order to test this very general hypothesis, new experiments will be required. They will take as their elements anatomical areas shown to be active by PET or MRI during simple computations. Subjects can then be asked to assemble these elements in different ways, and the dynamics of the circuitry can be checked as in our studies by the use of electrical methods sensitive to brief fluctuations in neural activity.
NETWORKS OF ATTENTION
Nearly a century ago, the great American psychologist and philosopher William James wrote about selective attention:
"Everyone knows what attention is. It is the taking possession by the mind, in clear and vivid form, of one out of what seems several simultaneously possible objects or trains of thought. "
Although we don't know exactly how focal awareness is constructed, we regard the networks involved in selective attention as basic to its construction. The last five years have seen a tremendous increase in our understanding of three networks, each of which carries out a function important to selective attention, such as orienting, detecting events, or maintaining the alert state. We believe that with a better understanding of how these networks function there will come an understanding of the brain systems that support conscious awareness. We suppose that the anterior cingulate is involved in the kind of reprogramming of mental operations discussed in Chapter 6. Recall from that chapter that we asked subjects to identify conjunctions, which were words with two specified features such as a thick letter and membership in a specified semantic category, in our case "is manufactured." We found that subjects could perform either the visual computation or the semantic computation first, depending on our instructions. We believe that the order of the computations is determined by the degree to which the computations are primed by the executive attention network. When priority is given to the semantic computation, the executive network activates lateral frontal areas and the semantic category is evaluated first. When priority is given to the visual feature, the executive network activates more posterior areas related to visual features, and the letter thickness is analyzed first. Obviously much needs to be learned about the mechanisms by which this priority is achieved by the executive system. So far we have merely drawn a kind of first sketch of the networks involved and developed some methods to measure their influences on distributed computations.
BRAIN DEVELOPS MIND
When an anatomical picture showing the organization of the primary visual cortex was created from the Nobel-prize-winning work of David Hubel and Torsten Wiesel, it spurred great interest among neuroscientists in understanding how visual mechanisms developed. For the most part, their studies used nonhuman animals rather than human infants since the evidence was gathered by manipulating the animal's visual environment and observing the effects at the cellular level. The findings outlined in Chapter 7 have provided a basis for understanding the anatomy of attention at the level of neural systems. Although research on human infants does not allow recordings at the cellular level, studies with infants could still give us an understanding of how these attention networks develop.
Even more interesting, studies of infants have a useful advantage: such studies could enable us to observe how attention regulates other cognitive and emotional systems. We are impressed with the striking changes in infant behavior over the first year of life. As the visual system matures during the first six months, infant behavior often appears dominated by visual orienting. Later, as executive control develops, we see behavior controlled by the infant's own internal agenda and observe the origins of language. Thus there is good reason to believe that the networks of local brain activity reviewed in the last few chapters play a role in the appearance of much human behavior.
For this reason it becomes important to determine if we can measure attention in infants and observe its development. To explore the development of the visual orienting network over the first year of life, we use tasks similar to those described in Chapter 7, in which the subject's attention is first drawn by a cue and then by a target. Infant's eyes are fixed on a central screen at the start of each trial, and two peripheral screens are used to attract their eyes. We record the eye movements on TV tape so that they can be analyzed frame by frame. We use these eye movements as the behavior to be measured, just as in adults the key press serves as an overt measure from which we infer covert shifts of attention.
How does the ability to control attention affect the life of the infant? A major problem of infancy is the control of distress. Caregivers provide a hint of how attention is used to regulate the emotional state off the infant. Earlier than 3 months after birth, caregivers report themselves as using mainly holding and rocking to quite their infants. However, at about 3 months many caregivers, particularly in Western cultures, attempt to distract their infants by bringing their attention to other stimuli. As infants attend they are often quieted and their distress appears to diminish. However, a systematic study conducted at the University of Oregon by Cathy Harman suggests that the loss of overt signs of stress is not always accompanied by a genuine loss of distress. Instead, some internal system appears to maintain the initial level of distress, and the overt distress returns if the infant loses its orientation to the novel event. A possibly related phenomenon occurs in adults. Adults who report themselves as able to focus and shift attention also say they are less prone to depression and anxiety than those who report themselves as less able to control their attention. Attention may serve to control levels of distress in adults in the way somewhat similar to what is found early in infancy. Indeed, many of the ideas of modern cognitive therapy are based upon the link between attention and depression or anxiety. For example, people in therapy for depression may be trained to attend to positive life events. And directing attention away from painful stimuli has long been known to lessen the experience of pain.
MENTAL DISORDERS
Mental illness can be extraordinarily frightening. How can the mind treat a part of its own body as belonging to someone else, believe itself controlled by an alien force, or want to destroy itself? There is a long-standing belief that mental illness is unlike the physical illnesses of the body or even of the brain. Yet strokes and tumor growth in the brain also lead to frightening changes in mental experience. In these cases, however, it's easy to attribute changes in mental experience to brain damage because the lesion can be seen when the brain's anatomy is imaged. But what about the illnesses more commonly designated as mental, such as, for example, depression or schizophrenia?
DEPRESSION
Everyone feels unhappy now and then; a person suffering from major depression, however, experiences recurrent episodes of deep depression, which are separated by period of normal mood. Rather than simply consisting of sadness, the depressed mood may be described as an emotional "pain" or an inability to experience pleasure - sometimes referred to as anhedonia. While in this persistent emotional state, a person may well lack energy and become less active . He may have trouble sleeping or, conversely, sleep more than usual, and his appetite may change. Even cognition is affected: the symptoms of major depression include persistent ruminations of guilt, failure, and inadequacy. The person may think frequently of death or of committing suicide. In contrast to the normal sadness experienced at the loss of a loved one, for example, major depression does not simply arise in response to an obvious stress, and it diminishes when treated with antidepressant drugs.
Major depression is not a single disorder, but probably a group of disorders, each possibly associated with a different brain abnormality. Two important categories of depression are recognized: the sufferer of bipolar depression swings between manic and depressive episodes, whereas someone with unipolar depression has only depressive episodes. A variety of studies have suggested but not defined a neurological basis for major depression. Would a PET study be able to shed additional light on the physical cause of this disorder? A team of investigators in the PET laboratory at Washington University set out to find the answers, led by psychiatrist Wayne Drevets. We studied patients with unipolar depression who also had a parent, sibling, or child with unipolar depression, but no family history of alcoholism, antisocial personality, or mania. This condition is known as familial pure depressive disease. We reasoned that studying such a homogeneous population would give us the best chance of unambiguously uncovering the neurobiological basis of depression. We were not to be disappointed.
In our initial studies, we obtained PET scans of brain blood flow from both a group of depressed patients and a control group of normal subjects of the same age who had been carefully screened for any evidence of psychiatric disease. The scans within each group were averaged together to produce one composite control scan of the normal subjects and one composite scan of the depression patients. The composite scan of the control subjects was then subtracted from the composite scan of the depressed patients to reveal the functional brain changes associated with depression. In doing so, we had extended our functional brain changes associated with depression. In doing so, we had extended our functional brain imaging strategy: from subtracting images from a single subject in order to isolate brain circuits concerned with specific but transient mental operations, we had moved to subtracting images across a group of subjects at rest in order to characterize the neurobiology of a stable but disordered mental state.
The images from this study immediately revealed striking increases in blood flow in the frontal cortex, especially on the left side. Conversely, decreases in blood flow appeared in the parietal and posterior temporal lobes. The increases were found in several areas already familiar to us from our studies of the executive attention systems and language, but now these areas were, surprisingly, participants in the emotional state of depression. They included only a portion of the anterior cingulate but also the lateral surface of the frontal lobe, including part of the area active in the generation task discussed in Chapter 5.
The decreases in blood flow were particularly noticeable in areas associated with the attention network for orienting. Perhaps in major depression the neural systems involved in processing external information and maintaining alertness are suppressed in favor of systems involved in processing internally generated information such as thoughts and emotions.
Further study revealed additional areas whose activity distinguished normal individuals from those with depression. These included a group of cells located deep within the anterior portion of the temporal lobe and known as the amygdala because of its almond shape; a portion of the thalamus known as the medial dorsal nucleus; and an area in the basal ganglia known as the caudate nucleus. Blood flow increased in all these areas except the caudate nucleus, where it decreased.
What were we to make of these changes? Did these areas have anything in common other than a collective change in their activities in patients with depression? The answer emerged when we considered the connections linking these areas to other regions of the brain, which were well known from extensive studies in animals. The first circuit can be thought of as a triangular circuit in which the amygdala and the prefrontal cortical regions are interconnected with each other and with the thalamus. Each area in this circuit, when activated, stimulates the other areas and is stimulated in turn, as indicated by the arrows and the plus signs in the diagram. The second circuit is actually an indirect side loop from the prefrontal cortices and the amygdala to the thalamus; it permits both the prefrontal cortical areas and the amygdala to further enhance the activity of the thalamus by way of a complex chain of connections through the caudate nucleus and other areas of the basal ganglia, including the globus pallidus.
Within the above circuitry the caudate nucleus and the frontal cortices are modulated by dopamine released from cells residing in the midbrain. An increase in dopamine in these areas results in a decrease in their activity. Thus, a dopamine deficiency could account for some of the changes we observe in depression. However, abnormalities at other sites within the two circuits could also be the cause. From this initial study we could not identify the underlying abnormality. For the answer we turned to a second study. A group of patients with a history of episodes of severe depression were studied while they were in remission; all had been diagnosed as having familial pure depressive disorder. At the time of this study their mood was entirely normal and they were not receiving any medication. The results were dramatic. All the areas abnormal in the depressed group were now normal with the exception of the amygdala. Thus the amygdala had emerged as the trouble spot in this form of depression: something about this structure had predisposed these individuals to recurrent bouts of depression. A third study reinforced this impression. Patients on antidepressant medication and in remission had normal blood flow in the amygdala.
It is not surprising that the amygdala might play such a central role in a disease like depression. That structure plays a major role in assigning emotional significance to the life experiences that we hold in long-term memory. A dysfunction of the amygdala could cause a negative emotional state to be inappropriately assigned to all experiences, resulting in a depressed mood, or it could prevent positive emotional labels from being assigned to any event, resulting in anhedonia.
How were we to understand the role of the other areas active in depression? We were especially curious about those such as the left prefrontal cortex and the anterior cingulate cortex, which we had encountered previously under seemingly unrelated circumstances. It was obvious that depression, at least familial pure depressive disorder, was not represented in the brain by a unique set of areas exclusively devoted to emotion. Some of its consistent operations were clearly shared with other types of behavior.
Blood flow is elevated in the left prefrontal cortex during a bout of depression, but we have also seen that blood flow is elected in that same area when unpracticed subjects generate verbs in response to nouns. Could that area be carrying out a similar function in both cases? Further insight came from asking normal subjects to contemplate sad thoughts or memories to induce a sad mood. During the induced sadness, blood flow increased in the left prefrontal cortex, the same location where blood flow was elevated in depressed patients with familial pure depressive disorder. Both when contemplating sad thoughts and when generating verbs, subjects consciously associate thoughts or emotions with information held in long-term memory. The association is accomplished in short-term representational memory, also known as working memory, to which the prefrontal cortex is thought to contribute. Depressed subjects may engage in a similar associative activity as they experience incessant negative thoughts. Like word generation and thinking sad thoughts, depressive ruminations involve making conscious associations in working memory with information retrieved from long-term memory: these associations are semantic in the case of the word generation task, and emotional in the case of the negative thoughts in major depression and induced sadness.
Patients with brain damage to the prefrontal cortex often persist in strategies that have become inappropriate; during the Wisconsin card sorting task, for example, they continue sorting according to a criterion that they know is incorrect. It is of interest that patients with major depression describe their ruminations as intrusive and exceedingly difficult to discontinue. A dysfunctioning left prefrontal cortex appears to interfere with the ability of subjects to shift both emotions and thoughts.
The cingulate cortex was also active in depression and induced sadness. In this case, however, the area active in depression differed from the area active in the word generation task. The area active in word generation was located in the anterior cingulate just before it curves forward and down around the front end of the corpus callosum. In depression the activity was confined to the area of the cingulate just after it curves, which has been associated more closely with emotion in animal studies. Interestingly, induced sadness produced activity in both areas of the cingulate cortex, probably because the processing requirements overlap with both word generation and depression.
Whether depressed, merely sad, or absorbed in the task of generating words, subjects showed increased blood flow in the left medial thalamus. In the amygdala, however, blood flow was increased only in the depressed subjects. In contrast, blood flow was significantly decreased in the amygdala during both the generation of words and the contemplation of sad thoughts. Clearly the circuits that we had identified through our studies of depressed patients were operating differently under the conditions of these two other tasks. There is now the real possibility of discovering the cognitive operations involved in depression and the neural systems responsible for their implementation. In so doing, we will have achieved an understanding of an important emotional component of our lives, be it normal sadness or severe depression. Cognitive science and neuroscience, together, may well provide us with a clearer explanation of the neurobiology of depression and other emotions.
FUTURE IMAGES
In this final chapter we try to imagine the future. Unlike the images created by PET and MRI, our images of the future are of uncertain resolution. We can only speculate based upon what we now think about the physical embodiment of the mind. The obvious truth that there is much we do not know should not obscure the long way scientists have come in this effort, and methods such as those discussed in previous chapters offer still more opportunities for developing our knowledge. In this chapter we speculate on what may emerge from the next period of cognitive neuroscience, in method, in the integration of different levels of inquiry, in the study of individual differences, and in theories of mind and brain.
One of the most encouraging results of this work is that it has provided strong reasons for the continued development of new imaging methods. After all, there are only a limited number of highly inventive people, mostly in the physical sciences, who have the skills to develop methods of imaging the human brain, and they have the choice of many questions on which they can fruitfully spend their time. It remains important to detect atomic particles, to probe the distant reaches of the universe, and to sequence the human genome, but we believe that to this list can not be added the goal of improving the ability to see thoughts. The attempt to see more, and with higher temporal and spatial resolution, is likely to remain an attractive frontier of instrumentation.
INDIVIDUAL DIFFERENCES
One of the most surprising outcomes of the research described in this book is the ability to average images across different people to arrive at a composite representation of the operations being studied. This ability has led us to emphasize commonalities among people in the use of brain areas. It is well known, however, that human brain anatomy sometimes differs considerably from one person to the next. Studies of monkeys suggest that there can be a considerable distance between the location of the same functional area in different brains, and, in addition, many familiar anatomical landmarks may be missing entirely in some normal brains. Furthermore, cognitive studies have long established that individuals may differ in strategies they apply to even relatively simple tasks like reading or remembering words.
There have already been some practical applications of even this crude method of imaging individual mental operations. Neurosurgeons need to know if a patient's anatomy differs from the norm so that they don't inadvertently damage a vital area during surgery. For example, although the speech function is in the left hemisphere in most people, a minority of left handers have speech represented primarily in the right hemisphere. The standard way for studying the anatomy of an individual's language function before neurosurgery has been to administer a test called the Wada test, in which sodium amytal is injected into the carotid artery. When the drug is injected on the side that represents language, speech is arrested. This test entails both risk and discomfort for the patient, however. Moreover, the test can only give information about speech, which of all the language functions is the one least confined to a single hemisphere. PET offers a new way to determine a patient's anatomy. A test has been conducted by Jose Pardo and Peter Fox in St. Louis, using the tasks described in Chapter 5, to compare
PET with the Wada testing as a way of imaging the locations of different language functions. The nine persons who participated in the test were candidates for neurosurgery to reduce the severity of their epileptic seizures. In eight of the nine subjects, the PET and Wada testing agreed on the hemisphere of speech function, but in addition PET provided information on the sites of language functions that could not be obtained by the Wada test. In the one patient about which the two methods disagreed, subsequent neurosurgery based on the standard Wada data produced a long-lasting language deficit. These results suggest that PET or other neuroimaging methods will become important tools for localizing psychological functions before brain surgery.
In cognition, the study of the acquisition of knowledge systems underlying "expert" behavior has been, as previously noted, one of the most active areas of the field. Herbert Simon has argued that 50,000 hours or more of practice allows that chess master to develop a highly elaborated semantic memory structure necessary to play at the highest level. This semantic structure allows the master to reproduce, after a mere glance, the more than 30 pieces that occupy the chess board in the middle stages of a master-level game. Even a good, but not expert, chess player comes nowhere near this level of performance. Yet the memory of the master for a random chess board configuration or for other kinds of information is only average. Practice has given the master player a way of representing information that comes into play rapidly and effortlessly.
How has the brain of the chess master changed? The simple answer is that we do not know, but we now have some models that allow us to begin to work on understanding the acquisition of expertise. Most people in our society become experts in reading English words. We discussed brain processes underlying this skill in some detail in Chapters 5 and 6. One of the most remarkable facts about this skill is that its development produces a part of the visual system that is sensitive to English words and strings of letters that obey the rules of English orthography, but is not sensitive to consonant strings. Using noninvasive scalp electrodes, it is now possible to observe the development of this level of expertise in children as they acquire literacy and to compare their brain activity with that found in expert readers.
TOWARD A THEORY OF THE BRAIN?
It seems appropriate that a book such as this should end with a theory of the brain. Such theories are common these days. They tend to stress principles at one level, often the cellular level, and derive most of human psychology from them. Our approach has been rather different. We have instead tried to provide a framework that allows us to relate research at many levels.
A theory of cognitive neuroscience would tell us how the brain works. It would be integrated in two senses. It would give an account at all the levels described in our general framework, from a specification of the cognitive systems to an understanding of the cellular mechanisms that support them. It would also describe how these cognitive systems achieve the subjective conscious experience that we call "mind". Such a theory would be a solution to the philosophical separation between mind and brain that was expressed in the theory of Rene Descartes.
We do not think that such an integrated theory is at hand, at least not in our hands. Yet, there is a sense in which the images of mind that we have described have rendered the mind-brain separation obsolete, at least at the everyday working level at which science is done. In our laboratories we describe the shift of attention, the visual word form, or the system responsible for target detection at the cognitive level - that is, we describe a sequence of mental operations. We design and execute experiments to see where in the brain these operations occur. We sometimes make correct predictions and sometimes find out new things, but each experimental design goes from the cognitive to the anatomical or the reverse. Experiments in research centers throughout the world now move effortlessly between the description of mind and the anatomy of brain as though there had not been the centuries of philosophical disputation about whether it is even possible.