WORKING MEMORY RETENTION SYSTEMS: A STATE OF ACTIVATED LONG-TERM MEMORY

To be published in Behavioral and Brain Sciences (in press)
© Cambridge University Press 2002

Below is the unedited, uncorrected final draft of a BBS target article that has been accepted for publication. This preprint has been prepared for potential commentators who wish to nominate themselves for formal commentary invitation. Please DO NOT write a commentary until you receive a formal invitation. If you are invited to submit a commentary, a copyedited, corrected version of this paper will be posted.

Working Memory Retention Systems: A State of Activated Long-Term Memory

Daniel S. Ruchkin

University of Maryland

School of Medicine

Department of Physiology

Program in Neurosciences

Baltimore, MD USA

Jordan Grafman

National Institutes of Health

Cognitive Neuroscience Section

NINDS

Bethesda, MD USA

Katherine Cameron

Washington College

Department of Psychology

Chestertown, MD USA

Rita S. Berndt

University of Maryland

School of Medicine

Department of Neurology

Program in Neurosciences

Baltimore, MD USA

Short Abstract: Electrophysiological studies of short-term storage in working memory indicate that there is 1) sustained co-activation of pre-frontal cortex and posterior cortical systems involved in the initial encoding of the retained information, 2) an increase in neural synchrony between pre-frontal cortex and posterior cortex, and 3) enhanced activation of the long-term memory representations of the material maintained in working memory. A parsimonious interpretation of these findings is that activation of long-term memory associated with posterior cortical encoding systems provides the representational basis for working memory. In this view, there is no reason to posit specialized neural networks whose functions are limited to that of short-term storage buffers.

Abstract: High-temporal resolution event-related brain potential (ERP) and electroencephalographic (EEG) coherence studies of the neural substrate of short-term storage in working memory indicate that the sustained co-activation of both pre-frontal cortex and the posterior cortical systems that participate in the initial perception and comprehension of the retained information are involved in its storage. These studies further show that short-term storage mechanisms involve an increase in neural synchrony between pre-frontal cortex and posterior cortex, and enhanced activation of the long-term memory representations of the material held in short-term memory. This activation begins during the encoding/comprehension phase and evidently is prolonged into the retention phase by attentional drive from pre-frontal cortex control systems.

A parsimonious interpretation of these findings is that the long-term memory systems associated with the posterior cortical processors provide the necessary representational basis for working memory, with the property of short-term memory decay being due to, primarily, the posterior system. In this view, there is no reason to posit specialized neural systems whose functions are limited to that of short-term storage buffers. Pre-frontal cortex provides the attentional pointer system for maintaining activation in the appropriate posterior processing systems. Limitations on the number of pointers that can be sustained by the pre-frontal control systems determines short-term memory capacity and phenomena such as displacement of information in short-term memory.

Keywords: Coherence; Event-Related Potentials; Short-Term Storage; Verbal; Visuo-Spatial; Working Memory

1. Introduction

1.1. History of the working memory model

Working memory refers to the collection of cognitive systems thatmaintain task-relevant information in an active state during the performance of a task. It functions as a "work space" in which recently acquired sensory information and information from long-term memory are processed for further action (e.g., storage, computation, decision-making). The construct of working memory evolved from previously developed models of memory systems that postulated a distinct short-term store, such as Atkinson and Shiffrin’s (1968) modal model. In the moda1 model the short-term memory system received input from sensory stores and transferred information to and from long-term stores. While the modal model accounted for a number of empirical results, it did not provide an accurate account of how short-term and long-term memory interacted, did not correctly predict performance for certain dual task experiments, and did not provide an adequate explanation for the memory performance of amnesiacs. In their seminal and influential model of working memory, Baddeley and Hitch (1974) resolved many of these short-comings by postulating a multicomponent working memory system consisting of a central executive that controlled conscious processing, with access to a pair of subsystems that temporarily stored phonological and visuo-spatial information. Baddeley (2000) recently revised this model, postulating a third short-term storage subsystem, an episodic buffer that forms an interface between the short-term phonological store, the short-term visuo-spatial store, and long-term memory. The episodic store augments working memory storage capacity, holding integrated material such as scenes and events. The central executive is regarded as a controller of deployment of attention, with no storage capacity. A key aspect of Baddeley’s model is that the various subsystems draw upon different processing resources and can, to some extent, function independently of each other.

A number of other models of working memory have been proposed since the initial Baddeley and Hitch (1974) paper. Although the models differ, most, but not all, share Baddeley’s view of multiple subsystems and temporary stores based upon modality-specific codes (see Miyake and Shah (1999) for discussions and comparisons of various current models of working memory). An important distinction among the various conceptualizations of working memory is in how short-term storage is implemented and its relationship to long-term memory. Baddeley posited that the working memory short-term storage modules are separate from long-term memory storage modules (1986; 2001; 2002). This view is based upon neuropsychological dissociations between performance on tasks involving either primarily short-term memory or primarily long-term memory resources. Baddeley (1999) suggested that information not accommodated by the working memory short-term stores (e.g., lexical and semantic information in verbal working memory) contributes to working memory performance via activation of representations in long-term memory. Hulme et al. (1997) and Saint-Aubin and Poirer (1999) further articulated this idea, proposing that lexical and semantic contributions to serial recall in verbal short-term memory are via a redintegration process that reconstructs degraded phonological codes during retrieval.

1.2. Short-term memory as activated long-term memory

Investigators such as Crowder (1993) and Cowan (1995; 1999; 2001) have been proponents of a contrasting view of short-term memory operation, namely that “long-term memory” and “short-term memory” are different states of the same representations, with activated representations in long-term memory constituting all of short-term memory. Based on findings such as the occurrence of serial position and recency effects in both short-term and longterm memory tasks, Crowder (1993) argued that short-term and long-term memory follow similar rules, and hence there was no reason to postulate separate long- and short-term storage systems. In Crowder’s view, memory storage takes place in the same neural structures in which the information was initially processed. Fuster (1995; 1997) has taken the same proceduralist position, based upon observations of single neuron activity in primates during short-term memory tasks. Fuster (1995) commented that we are dealing with “the memory of systems, not ... systems of memory.”

Cowan’s (1988; 1995; 1999) views are similar to Crowder’s, namely that short-term memory stores are constituted by an activated subset of long-term memory. Cowan argues for the construct that short-term memory involves all information accessed by a task, including 1) activated memory in the focus of attention, 2) activated memory not in the focus of attention, and 3) inactive memory accessible by activated retrieval cues. Short-term auditory sensory memory processes in experiments involving presentation of multiple streams of stimuli are examples of the latter two types of activation (Cowan, 1984; Cowan, 1995). Deployment of attention is a crucial feature of Cowan’s model of short-term memory, with attention sustaining and limiting activation of long-term memory. In Cowan’s model the capacity limitation of short-term memory is due to the limited capacity of the focus of attention (1999).

With respect to the role of sensory information in short-term memory, Penny (1989) has hypothesized that verbal short-term memory involves, in addition to phonological codes, contributions from modality-specific auditory and visual codes. A number of lines of evidence support Penny’s view: 1) memory is improved when different items are presented in different modalities than when all items are presented in the same modality; 2) recall is enhanced when items are organized by modality than by time of presentation; 3) two concurrent verbal tasks can be more effectively performed when different input modalities are utilized in comparison with only one input modality.

Baddeley (2001) claimed that construing short-term memory as activated long-term memory is inconsistent with neuropsychological data, since there are individuals with long-term memory deficits but not short-term memory deficits, and individuals with short-term memory deficits but not long-term memory deficits. However, Cowan (1999) has argued that long-term memory deficits are not necessarily due to damaged stores. Rather, such deficits can be due to impaired binding processes involving hippocampal-neocortex connections responsible for eliciting simultaneous activations across long-term stores that lead to an episode being stored (Rickard & Grafman, 1998). Activation of individual stores that accompanies short-term retention of information can be preserved in amnesia.

With regard to patients with short-term but not long-term memory deficits, Vallar and Baddeley (1984) reported an individual whose span in verbal serial recall tests was below the normal range, but whose performance on word learning, paired-associates learning and short-story learning tests was within the normal range. The poor span in serial recall was attributed to an impaired phonological short-term store. While Baddeley (2001) interprets these results as evidence for distinct short-term and long-term phonological stores, the normal performance on the learning tests of long-term memory may have been due to lexical, semantic and syntactic processes invoked by the learning tasks that compensated for the impaired phonological processing. Nevertheless, if verbal short-term memory representations are activated verbal long-term memory representations, then deficits in verbal short-term memory for specific types of representations should be indicative of impairments in establishing long-term memories for those representations. Romani and Martin (1999) have reported that individuals with a semantic short-term memory deficit also have problems forming semantic but not phonological long-term memories, whereas individuals with a phonological short-term memory deficit show the reverse pattern of difficulty. Thus, when the nature of the representations are taken into account, the neuropsychological evidence for distinct short-term and long-term memory stores is not compelling.

1.3. Episodic memory

While activation of long-term memory representations of items contributes to their retention in short-term memory, it does not account for all aspects of the retention process. Serial recall of the order of events involves conjunctions of representations that are not likely to be retained and recalled by activation of the representations alone. Both Cowan (1995; 1999; 2001) and Baddeley (2000; 2001) proposed that retention of serial order information involves the formation of new episodic links between the activated representations of the items held in short-term memory. Baddeley (2001) further argued that there is a distinct short-term store for episodic links, citing a fMRI study by Prabhakaran et al. (2000) as providing neural evidence of such a store. Prabhakaran et al. (2000) found greater activation in right prefrontal cortex when integrated (i.e., requiring binding of individual items) rather than unintegrated visual stimuli were used in a short-term memory task. The authors’ claimed that this was evidence for a ‘buffer’ for the temporary retention of integrated information. We suggest that a more precise interpretation of the fMRI data is that the right prefrontal cortex participates in the process of maintaining binding information in an active state. The question of whether there is a separate store for binding information, or whether the binding ‘representations’ are based upon the same neural structures that, with consolidation, become part of the long-term memory representations for the bindings, was not resolved by Prabhakaran et al.’s (2000) findings. Our view is that the neural connections underlying the binding processes that produce episodic links are the basis for both shortterm and long-term episodic memory. Recall and maintenance of episodic information involves activation of the binding ciruitry, while retention of novel episodic information involves the operation of binding formation and the initial consolidation process. In either case, the same neural connections are involved.

1.4. Scope of reviewed research supporting activation models of short-term memory

In contrast with Baddeley’s claim that short-term and long-term memory memory stores are distinct, this article will argue for the view that short-term memory corresponds to activated long-term memory and that information is stored in the same systems that initially processed the information. On theoretical grounds, activation-proceduralist models have the advantage of parsimony with respect to models that postulate distinct short-term and longterm memory stores. On empirical grounds, there are electrophysiological and hemodynamic imaging data, reviewed in this article, from normal, intact humans that substantiate activation models.

Research into short-term memory has utilized cognitive experiments, studies of patients, and functional neuroimaging techniques to motivate an understanding of how information is retained over short periods of time and which brain areas are crucial for encoding, retaining, and retrieving information held in short-term memory. What has been lacking in these studies is accurate information regarding the timing and duration of the various processes enlisted in short-term memory operations. Hence, in this paper, we will concentrate upon studies that employed high temporal resolution event-related potentials (ERPs) to provide information about the timing of brain processes involved in short-term memory operations. These studies have provided unique and novel information on the mechanisms employed in active maintenance and how networks involved in short-term storage operations map onto networks involved in the perception, encoding and determination of meaning. We will also review hemodynamic imaging studies that have provided key anatomical information that complements the ERP findings. The results of these varies studies are most compatible with models of short-term storage operations that stress sustained activation of the perceptual and associated long-term memory systems involved in the initial bottom-up processing of information, and posit an important role for attentional systems in the maintenance process (e.g. Cowan (1999)), as opposed to models that stress buffers based upon neural systems that are specialized for short-term storage (e.g. Baddeley (1999)).

Lesion and hemodynamic imaging investigations have produced abundant converging data on the location of brain regions that contribute to short-term memory operations (Vallar, De Betta, & Silveri, 1997; Vallar & Papagno, 1995; Cabeza & Nyberg, 1997; Smith & Jonides, 1999; Cabeza & Nyberg, 2000). The data have consistently supported models of short-term memory that are based upon multiple subsystems and modality-specialized temporary stores. In this context, we note that findings of multiple, modality-specialized short-term stores are fully compatible with the position that short-term memory corresponds to activated long-term memory representations, given that longterm memory involves multiple, modality-specialized stores.

Tasks that entail manipulation of information and updating memory, functions of the postulated central executive, evidently involve multiple sites in frontal cortex (D'Esposito et al., 1995; Manoach et al., 1997; Postle, Berger, & D'Esposito, 1999; Owen, 1997). Encoding and storage of phonological information involves left parietal and left frontal regions that underlie language processing and speech production (Paulesu, Frith, & Frackowiak, 1993; Awh et al., 1996; Awh, Smith, & Jonides, 1995; Jonides et al., 1998; Henson, Burgess, & Frith, 2000), while encoding and storage of visuo-spatial information engages ventral (inferior temporal cortex) and dorsal (posterior parietal cortex) visual processing pathways involved in perceptual processing (Jonides et al., 1993; Courtney, Ungerleider, Keil, & Haxby, 1996; Courtney, Ungerleider, Keil, & Haxby, 1997; Haxby, Petit, Ungerleider, & Courtney, 2000; Awh & Jonides, 2001). Such studies have been very useful in mapping the cognitive architecture of human short-term memory to specific brain regions. However, the capability of hemodynamic imaging to provide direct, detailed information on the timing of neural processing underlying the operation of short-term memory is limited, since hemodynamic responses can be substantially delayed and prolonged in comparison with neural and behavioral responses.

The complementary approach of using high temporal resolution techniques such as electroencephalographic (EEG) and/or magnetoencephalographic (MEG) recordings from scalp can be useful when dealing with timing issues. Although the capability for determining the locations of brain activation from scalp recordings is limited, EEG and MEG recordings can provide real-time measures of brain activity with sub-millisecond accuracy. ERPs extend knowledge gained from hemodynamic imaging studies by providing a detailed tracking of the time-course of brain activation across encoding, retention, and retrieval phases of an eliciting event. Long duration, sustained processes and short, limited duration transient processes can be distinguished with ERPs. In addition, through the comparison of rapid fluctuations in EEG recordings from multiple sites, EEG coherence analysis provides information on the degree of neural synchronization between brain regions within a specific time interval and frequency band. EEG coherence measures provide an approach for investigating interactions among brain regions during short-term memory operations, thereby further extending knowledge gained from ERP studies. Issues such as whether cognitive processes that operate in parallel interact can be best addressed by coherence methods.

The ERP studies considered below usually employed either delayed matchto-sample or delayed serial recall paradigms, with delay intervals in the range of 3000 to 4000 ms. The strategy in these studies was to manipulate the type and/or amount of information held in short-term memory and test whether the manipulation produced differences in brain activity during the delay interval. Differences in the delay interval of ERP timing and/or amplitude as a function of the information held in short-term memory indicate that brain activity during retention is sensitive to such information. Finding that brain activity during retention is influenced by the type of information held in short-term memory is interpreted as evidence that the information is being held in an active state during retention. Differences between conditions in ERP scalp topography further indicate that the anatomical configuration of the generators of the brain activity differs between the conditions. (A brief discussion of topography and estimation of the brain sources of scalp recorded ERP activity is presented in the Appendix.) Consequently, variations in the delay interval of ERP topography with the manipulation of information maintained in short-term memory are interpreted as evidence that the configuration of brain systems active during retention varies with the nature of the information. Coherences between recording sites reflect the pattern and degree of connectivity between brain regions. Thus, differences between coherences as material maintained in short-term memory is manipulated are interpreted as evidence that connectivity between brain regions active during retention is sensitive to the properties of the maintained material.

The results of the ERP studies reviewed below indicated that short-term retention processes involve sustained activation of both frontal cortical control systems and posterior cortical systems involved in perception and comprehension of visuo-spatial and linguistic information, with enhanced neural synchrony between the frontal and posterior systems during retention. They further indicated a greater diversity and specialization of retention processes than originally proposed by Baddeley and Hitch. For visual stimuli, in addition to the demonstration of separate sustained storage systems for visual-object and visual-spatial material, there is also evidence for transient, intermediate duration storage systems. For language stimuli, in addition to phonological codes, there is evidence that lexical-semantic and modality-specific codes actively contribute to the retention process, as opposed to only during recall by redintegration. The temporal morphology of the ERPs indicated that brain regions active during initial processing (prior to the retention interval) remain active during the retention interval, supporting proceduralist-activation models of memory proposed by Crowder (1993), Fuster (1997) and Cowan (1995; 1999). The behavior of an ERP deflection sensitive to priming indicated that consciously maintaining items in memory raises the level of activation of the long-term representations of the items to above the level reached by priming due to processing the items but not consciously holding them in memory. This finding provides strong support for the notion that activation of long-term memory representations is the root of short-term memory performance. We also review complementary hemodynamic imaging studies that provide anatomical support for proceduralist, activation models and the idea that posterior cortex provides the representational basis for most short-term memory operations while pre-frontal cortex provides the attentional control.

2. Retention of visuo-spatial information in short-term memory

Numerous studies have shown that the visual system involves, beyond primary visual cortex, separate cortical pathways for the perception of object (ventral pathway) and spatial (dorsal pathway) information (Grady et al., 1992; Hanley, Young, & Pearson, 1991; Harter & Aine, 1984; Mangun, Hillyard, & Luck, 1993; Rosler, Heil, & Hennighausen, 1995; Ungerleider & Mishkin, 1982; Van Essen, Anderson, & Felleman, 1992). Both pathways include extra-striate cortex. The ventral pathway also includes inferior temporal cortex. The dorsal pathway also includes posterior parietal cortex. Results from hemodynamic (Smith et al., 1995) and ERP (Mecklinger & Pfeifer, 1996; Ruchkin, Johnson, Jr., Grafman, Canoune, & Ritter, 1997b) studies indicated that visual short-term memory divides along similar lines. However, the fMRI studies suggested that only perceptual and transient storage operations occur in posterior visual processing pathways (Haxby et al., 2000), while sustained storage occurs in pre-frontal cortex. In contrast, the ERP studies indicated that both transient and sustained storage operations occur in the posterior visual processing pathways. The ERPs showed that during retention there is sustained activation in brain regions underlying posterior and temporal scalp, with the amplitude of the sustained brain activity varying directly with memory load.

These results were obtained in delayed-match-to-sample tasks with linear arrays of geometric objects (object task) and two-dimensional patterns of randomly placed squares (spatial task) (Mecklinger & Pfeifer, 1996), or with schematic faces (object task) and the motion of an asterisk (spatial task) (Ruchkin et al., 1997b). The topographies of the scalp ERP activity in the retention interval of the Ruchkin et al. (1997b) study were sharply focused over parietal scalp in spatial tasks and more broadly distributed over parietal-to-frontal scalp in the object tasks (see Fig. 1), consistent with the sources of the scalp ERP activity during retention primarily involving the dorsal pathway for spatial information and the ventral pathway for object information. It has been further demonstrated that the patterns of ERP activity found in the retention interval of visual short-term memory tasks did not occur in control tasks that involved similar encoding and response processing, but had negligible memory demand (Ruchkin, Canoune, Johnson, Jr., & Ritter, 1995; Low et al., 1999).

Figure 1. Estimated current source density (CSD) maps for the object and spatial tasks of the scalp topography of ERP activity at the end of the retention interval (3010 ms to 3550 ms after stimulus offset). The difference between contour lines corresponds to a current density increment of 1µV/cm2. The current source densities were derived from the across-subjects averaged ERP amplitudes. In this figure and subsequent figures with maps, The shaded areas of the maps indicate positive amplitudes and the unshaded areas indicate negative amplitudes. The maps are 90_ projections with the front of the head at the top. Electrode positions are indicated by the dots. The vertical line of three dots in the center of the map correspond (from top to bottom) to midline frontal, central and parietal scalp sites, respectively.

Note the differences along the mid-line between the CSD maps for the object and spatial tasks. For the spatial task, the CSD has a pronounced negative focus over parietal scalp. For the object task, the CSD negativity is broadly distributed from parietal to frontal scalp. This topographic difference indicates that the configuration of brain sources active during retention is different in the object and spatial tasks.

Estimates of the locations and time courses of the brain sources of the scalp recorded ERP activity (Scherg, 1990) were used to examine the timing of activation in specific brain regions during the encoding and retention of visual object and spatial information (Ruchkin et al., 1997b). The time courses of activation in primary visual cortex, posterior and anterior temporal lobes, posterior parietal cortex, and pre-frontal cortex are illustrated in Fig. 2. The sources that best represented activity in primary visual cortex displayed early phasic responses, with maximal activation during stimulus presentation and relatively little activation in the subsequent retention interval (Fig. 2, top row). Brain regions active during the retention interval were mainly located in the ventral and dorsal visual processing pathways, and pre-frontal cortex. For both object and spatial information, the source analysis revealed a mixture of early phasic activity during stimulus presentation followed by long duration transient activity in the posterior temporal lobes (Fig. 2, second row from top). The long duration transients began during stimulus presentation and continued into the retention interval for about 2500 to 3000 msec. The source analysis further indicated that there was sustained activation in both pre-frontal cortex and posterior visual processing pathways during retention (Fig. 2, bottom two rows). For the object task, the sustained activity was in the anterior temporal lobes (ventral pathway), while for the spatial task the sustained activity was near the junction of parietal and occipital cortex (dorsal pathway). In the object task, where all the information to be retained was available at stimulus onset, sustained activity in the anterior temporal lobe started during stimulus presentation. The spatial task required memorization of a sequence of movements, with the last movement beginning 1500 ms after the stimulus began. In this case, sustained activity near the junction of parietal and occipital cortex did not begin until presentation of the stimulus sequence was complete. In both tasks, the onset of the sustained activity observed in the dorsal and ventral visual processing pathways was 60 to 300 milliseconds before the onset of the sustained activity observed in pre-frontal cortex.

The analysis of brain sources indicated that maintenance of visuospatial information involves sustained activation in both pre-frontal cortex and posterior visual processing systems. The finding of sustained activity in posterior cortex supports Cowan’s (1995) and Fuster’s (1997) views that maintaining activation in cortical regions subserving perception is a component of the retention process. Pre-frontal cortex and posterior perceptual regions interact, with pre-frontal cortex apparently providing the top-down control that extends activation in posterior cortex that began during perception and encoding.

2.1. Transient visual short-term memory

The sources with transient time-courses (Fig. 2, second row from top), brief windows of activity in the posterior temporal lobe presumably involved in intermediate visual processing operations, support Cowan’s (1995) contention that visual working memory consists of at least three stages, with a transient stage intermediate to an initial, high capacity, very limited duration iconic store that encodes the physical features of stimuli (Sperling, 1960), and a limited capacity, post-categorical sustained short-term store. It is noteworthy that the transient stores appeared to be in a part of the ventral pathway involved in preliminary processing of visual material, while sustained storage operations appeared to be in ventral or dorsal pathway regions involved in higher level processing of visual material.

2.2. Attention-based maintenance mechanisms

The view that short-term storage of visuo-spatial information depends, at least in part, upon enhanced activation in visual cortex due to attentionbased maintenance mechanisms, is supported by ERP studies of short-term memory and selective attention (Awh, Anllo-Vento, & Hillyard, 2000; Awh & Jonides, 2001). In the memory task, subjects remembered the locations of three falsefont characters, all of which were in either the left or right visual field. A probe stimulus presented during the delay interval elicited a short-latency phasic response that was larger when the probe was in the same visual field as the memory-set stimuli than when the probe was in the opposite visual field. The timing and topography of the enhanced response to the probe in the memory task was very similar to the enhanced response found in a visual selective attention task when the eliciting stimulus was in an attended location in comparison with when the stimulus was in an unattended location.

A combined PET-ERP source localization study (Hillyard & Anllo-Vento, 1998) has shown that enhanced ERP responses to stimuli in attended locations arise in extra-striate visual cortex, contra-lateral to the field of the attended stimulus. Furthermore, an fMRI study (Awh et al., 1999) that employed short-term memory and selective attention tasks similar to those in the Awh et al. (2000) ERP study, also found that hemodynamic activation was greatest in visual cortex contra-lateral to the field of the memorized or attended stimuli, with a high degree of overlap of fMRI activation in the memory and attention tasks. This convergence of results implies that shortterm storage of visual location information entails enhanced, sustained activation in cortical regions involved in the perception/encoding of the visual material, and the enhanced activation depends upon attention-based maintenance mechanisms.

Figure 2. Estimated time courses of activation in visual cortices and prefrontal cortex during encoding (pre-3000 ms) and retention1 (post-3000 ms) of visual-object or visual-spatial information. The time axis extends from 360 ms before to 5550 ms after stimulus onset. Stimulus duration is 2000 ms. The waveforms and their brain locations were estimated by source analyses (Scherg, 1990) of across-subjects (n=12) averaged ERPs recorded from scalp by a 24 channel montage. The vertical scale is in arbitrary units. In this and subsequent figures, stimulus presentation intervals are demarcated by vertical lines.

The source analyses indicate that primary visual cortex is most active during stimulus presentation, with relatively short latency phasic activity synchronized to the presentation of each stimulus (top row). There is sustained activity during retention in both pre-frontal cortex (bottom row) and the dorsal and ventral visual processing pathways (third row from top). The waveforms in the second row from the top further indicate that there is long duration transient activity that begins during stimulus presentation and then extends for 2000-3000 ms into the post-stimulus interval.

2.3. Summary: Visuo-spatial working memory

Implications of the ERP studies of visual working memory are summarized schematically in Fig. 3. The timing of the initial (less than 1000 ms poststimulus) phasic deflections in primary visual cortex and the posterior temporal lobes suggests that these brain regions contribute to the operation of the iconic store (Fig. 3, top row). The subsequent longer duration (_ 3000 ms) deflections in the posterior temporal lobes indicate the existence of transient, intermediate stores whose role may be to support the translation of information from iconic to sustained storage formats (Fig. 3, middle row). The sustained activity in the dorsal and ventral pathways indicates that short-term maintenance of visual information depends upon activation of posterior sensory processing systems as well as the pre-frontal cortex (Fig. 3, bottom row). This contrasts with fMRI studies of visual working memory by Haxby, Ungerleider and co-workers (Courtney et al., 1997; Haxby et al., 2000). In their fMRI studies, it appeared that activation in posterior visual processing pathways had a pronounced transient character in comparison with clear sustained activation in frontal regions. Thus, Haxby et al. (2000) suggested that the role of these posterior regions was mainly in the domain of perceptual processing, while short-term storage depended primarily upon frontal regions. The ERP findings suggest that the fMRI measures in posterior cortex may have given too much weight to the activation of the transient stores, thus cloaking the sustained activity in posterior cortex and its contribution to short-term retention of visual information. The role of the frontal regions may be in the domain of sustained attentional drive directed at those posterior regions whose activation is to be maintained.

3. Retention of verbal information in short-term memory

A number of ERP studies of delayed serial recall indicate that verbal short-term memory depends upon more than phonology during retention and redintegration of degraded phonological representations at retrieval. Utilizing sustained ERP activity recorded during the retention interval, Lang et al.(1992) and Ruchkin et al. (1997a) found that retention of verbal material involves processes that are specific to the modality of presentation, while Ruchkin et al. (1999) found that sustained supramodal lexical and semantic processes also were active during retention. Finally, an item recall study (Cameron, Haarmann, Grafman, & Ruchkin, 2002) indicated that the contribution of semantic representations in long-term memory to short-term retention was not simply a result of their being primed during study of the stimuli to be memorized. Rather, the act of maintaining information in shortterm memory results in a concurrent heightened activation of long-term memory representations, beyond the level of activation caused by priming associated with the initial processing of the stimuli.

Figure 3. Schematic of the timing of the activation of three hypothesized short-term storage systems that contribute to the operation of visual shortterm memory. The duration of the initial, iconic stage is about 500-1000 ms (Sperling, 1960). Based upon the results of ERP studies, the transient store operates over the 500-4500 ms latency range, while the onset of the sustained store is dependent upon the timing of stimulus delivery. In Ruchkin et al. (1997b), the onset of the sustained store was approximately 500-800 ms after all the information provided by the stimulus had been delivered.

3.1. Modality-specific processing streams in verbal short-term memory

Penney (1989) has argued that, along with phonological rehearsal, auditory or visual modality-specific verbal short-term memory processes support retention, depending upon whether the material is heard or read, with the auditory processing stream being more durable than the visual processing stream. The results of two ERP studies (Lang, Starr, Lang, Lindinger, & Deecke, 1992; Ruchkin et al., 1997a) support Penney’s contention. The pattern of ERP activity during the post-stimulus retention interval differed for verbal material (digits in Lang et al., a non-word in Ruchkin et al.) that was heard or read. Note the differences between the ERP waveforms (Fig. 4a) and scalp topographies (Fig. 4b) associated with the two modes of stimulation in the Ruchkin et al. (1997a) study.

Figure 4a. Across-subjects (n=13) averaged scalp ERPs in a delayed serial recall task in which the material was presented either aurally (dashed lines) or visually (solid lines). The task was to remember a pronounceable five syllable non-word. The time axis extends from 270 ms before to 5640 ms after stimulus onset. Stimulus duration was 2000 ms. In this and subsequent figures of scalp ERP waveforms, the ERPs were originally recorded with ACcoupled amplifiers (which attenuated low frequency ERP activity). The waveforms were digitally rendered to the approximate wave shapes that would have been obtained with DC-coupled amplifiers (no attenuation of low frequency ERP activity). The ERPs are plotted with negative polarity up with respect to a digitally linked A1 and A2 reference.

Note that for auditory stimuli there is a sustained frontal negativity, lateralized to the left, with a relatively short onset latency (during the stimulus interval). For visual stimuli, the sustained frontal negativity is lower amplitude, with an relatively late onset (after the stimulus interval). The ERPs elicited by the visual stimuli also display a transient positivity over centro-parietal scalp, that begins during stimulus presentation and ends about 2500 ms after stimulus offset, and a transient negativity, over bilateral posterior temporal and parietal temporal scalp. No such positivity is elicited by the auditory stimuli.

Figure 4b. Estimated current source density maps for the scalp topography of the ERP activity presented Fig. 4a. The maps are for the activity at selected time points in the retention interval: 500, 2000 and 3500 ms after offset of the 2000 ms duration stimulus. The difference between contour lines corresponds to a current density increment of 1µV/cm2.

Note that the maps for the auditory stimuli (upper row) indicate a relatively rapid build up of ERP negativity focused over left frontal scalp. In contrast, the maps for the visual stimuli (lower row) indicate that the left frontal negative focus builds up more slowly, while early in the retention interval there is a focus of positive activity over central-posterior scalp, and a bilateral focus of negativity over posterior temporal scalp that is not seen in the maps for auditory stimuli. These differences in timing and topography between the brain responses to auditory and visual stimuli are evidence for the contribution of modality specific processes to the operation of verbal short-term memory.

For visual but not auditory stimuli, Ruchkin et al. (1997a) found a long duration transient positivity at mid-line parietal and central sites that began during stimulus presentation and ended about 2500 ms after stimulus offset (Fig. 4a). The amplitude of the transient positivity increased directly with verbal memory load (Ruchkin, Johnson, Jr., Canoune, & Ritter, 1990; Ruchkin, Johnson, Jr., Grafman, Canoune, & Ritter, 1992; Ruchkin et al., 1994). The mid-line posterior transient positivity was not found in ERP scalp recordings obtained in visual-object and/or visual-spatial short-term memory tasks (155,523,685,812). In view of its timing, sensitivity to memory load and apparently exclusive elicitation by verbal material that is read, this aspect of the visual processing stream evidently indexes the operation of a visual, non-phonological verbal storage process. This process is possibly based on orthographic codes; it is active around the time of phonological recoding, and maintains representations of material that has undergone initial visual analysis (Shallice & Vallar, 1990; Vallar & Papagno, 1995).

There was sustained negative ERP activity in the post-stimulus retention interval for both auditory and visual stimuli. However, there were timing and topographic differences for the two modalities. For auditory stimuli, the sustained negativity appeared to have two constituents: 1) a left-lateralized negativity that was largest over frontal sites and negligible over posterior sites; 2) a lower amplitude right-lateralized negativity with roughly the same amplitude at frontal and posterior sites. Both of these auditory sustained negativities began during the stimulus interval. The negativity elicited by the visual stimuli also appeared to have two constitutents: 1) a frontal, left-lateralized negativity, with a lower amplitude and later onset (during the post-stimulus interval) than the left frontal auditory negativity; 2) a bilateral, transient negativity, most clearly discerned at posterior temporal sites (see Fig. 4a), that began during stimulus presentation and decreased over the post-stimulus, retention interval.

The left-frontal negativity that is common to both modalities (albeit with different amplitudes and onset latencies) probably reflects operations associated with maintaining phonological representations in verbal short-term memory. Its earlier onset for auditory stimuli is evidence for auditory material having a rapid, direct access to the phonological memory system while visual material first undergoes a more time-consuming re-coding to a phonological format before entering the phonological system through articulatory rehearsal (Shallice & Vallar, 1990).

The amplitude of the left frontal negativity varies directly with verbal memory load (Ruchkin et al., 1990; Ruchkin et al., 1992; Ruchkin et al., 1994), and there is a significant across-subject correlation between its amplitude and articulation rate (Ruchkin et al., 1994), suggesting that phonological rehearsal operations covary with the processing indexed by the left frontal negativity. A study that contrasted retention of familiar, verbalizable material with unfamiliar, non-nameable material indicated that the left frontal negativity incorporates a composite of executive control processes (Bosch, Mecklinger, & Friederici, 2001). Taken together, these various findings suggest that the left frontal negativity indexes a combination of attentional control and phonological rehearsal operations that are involved in the short-term retention of verbal material.

The ERP data map onto Penney’s view of verbal short-term memory. The left frontal negativity reflects sustained mnemonic operations, probably involving phonological representations, that are common to both auditory and visual stimuli. Modality specific operations are indexed by the sustained negativity over the right hemisphere elicited by auditory stimuli, and the posterior transient waveforms elicited by visual stimuli, namely the mid-line positivity and bilateral temporal negativity. The timing of the modalityspecific ERP patterns suggest that auditory verbal mnemonic processes are more durable than visual verbal mnemonic processes.

3.2. Contributions of lexical and semantic codes to retention

There is widespread agreement, based upon evidence from behavioral studies of intact and impaired subjects, that phonological codes are involved in the maintenance of verbal information in working memory (Baddeley, 1986). There is less agreement about whether lexical-semantic codes actively contribute to the maintenance process. One view is that lexical-semantic codes are not actively involved in retention. Rather, lexical-semantic information is thought to contribute to verbal working memory during retrieval, with lexical-semantic codes in long-term memory facilitating recognition of partially degraded information in the phonological store (Hulme, Maughan, & Brown, 1991; Hulme et al., 1997; Walker & Hulme, 1999). Alternative views stress that language processing activates a variety of codes (modality-specific, phonological, lexical, semantic, syntactic) that are maintained at different strengths, depending upon task demand, over time (Penney, 1989; Monsell, 1984; Saffran, 1990; Saffran & Martin, 1990; Martin & Romani, 1994; Martin & Saffran, 1997). While phonological rehearsal is a possible contributor to the maintenance of information in verbal short-term memory, it is viewed as neither necessary nor sufficient for all of the retention operations required of verbal short-term memory. Cowan and Kail (1988; 1996) postulated that the attention given to maintaining verbal material in working memory raises and prolongs activation of the words' longterm memory codes, and, through this enhanced activation process, lexicalsemantic codes contribute to retention of verbal information in working memory.

It is difficult to decide between these different conceptions of verbal working memory from behavioral data alone, since behavioral data reflect a combination of encoding, retention, retrieval and decision operations. Using the temporal resolution of ERPs, brain activity specific to the retention interval can be delineated and analyzed, so that the types of codes that influence the retention process can be determined. This approach has been applied across a series of studies that examined the contributions of lexical and semantic processes to brain activity during retention. These studies have shown that the patterns of brain activation during short-term maintenance of verbal material are influenced by the lexical status of the material (Ruchkin et al., 1999), 2) whether the referents of words are concrete or abstract (Ruchkin et al., in preparation).

3.3. Lexical status

Evidence for an active contribution of lexical codes to maintenance of verbal information in working memory was obtained from ERPs recorded during performance of a serial recall task involving retention of aurally presented words or pseudo-words designed to be maximally similar to the words in their sound structure (Ruchkin et al., 1999). The number of items in the word (five) and pseudo-word (three) lists were such that recall error rates were approximately matched for the two types of stimuli. The finding that five words could be retained at approximately the same level of accuracy as three pseudo-words was consistent with prior studies of verbal working memory (Hulme et al., 1991; Roodenrys & Hulme, 1993). It has been argued that the cause of the advantage of words over pseudo-words was that restoration during retrieval of partially degraded information in the phonological buffer was more effective for words (Roodenrys & Hulme, 1993). While such processing during retrieval may contribute to the word advantage, there is no compelling reason to believe that it is the only contributor.

Figure 5a. Across-subjects (n=11) averaged scalp ERPs in verbal short-term memory (left panel) and non-memory control (right panel) tasks contrasting the processing of words (solid lines) and pseudo-words (dashed lines). Stimuli were presented aurally. In order to approximately balance the error rates in the word and pseudo-word memory tasks, stimuli consisted of either five words or three pseudo-words. The time axis extends from 360 ms before to 8595 after word onset. To align the offset times of words and pseudo-words (5000 ms after word onset), pseudo-word onset was 2000 ms after the time of word onset.

Note that the sustained negativity during the post-stimulus retention interval in the memory task was larger for words. This effect was most marked in the vicinity of central mid-line scalp. There was no such difference between word and pseudo-word ERP activity in the post-stimulus interval of the non-memory control task.

Figure 5b. Maps of the scalp topography of the across-subjects averaged voltage fields for the ERP activity in Fig. 5a. The maps depict the distribution of ERP activity over the scalp at the end of the delay interval (3010 to 3500 ms after stimulus offset). The difference between contour lines corresponds to a voltage increment of .5 µV.

Note that the word and pseudo-word topographies display a marked difference over central scalp (more negativity for words) in the memory task (left column). The topographies in the control task (right column) are similar for words and pseudo-words, and differ from the topographies in the memory task. These results support the view that, when a conscious effort is made to hold words in short-term memory, lexical codes contribute to the maintenance process.

Lexical status also influenced ERPs during the delay interval (3600 ms) of the serial recall memory task, well after termination of stimulus presentation and well before retrieval commenced, indicating that brain activity during retention is directly influenced by the availability of lexical-semantic information (Ruchkin et al., 1999). Words elicited more negativity during retention than pseudo-words, with the effect being most marked at the central mid-line site (Fig. 5, left panel). This difference was sustained throughout the delay interval, with no indication of a significant increase with the approach of the time of retrieval. Thus, the effect of lexicality upon the ERPs reflected a process that subserved retention, rather than a retrieval-oriented process that developed during the retention interval. The effect of lexical status in the retention interval of the memory task was specific to consciously controlled memory operations, since lexical status had a negligible influence upon ERP activity in the poststimulus delay interval of a "non-memory" control task with similar attentional demands and stimulus and response processing requirements (Fig. 5, right panel).

Lexical status and number of items to be recalled (memory load) had different effects upon ERP activity during retention. Load was relatively high in the memory task (five or three items to be maintained in the delay interval) in comparison with the control task (one item, either "yes" or "no", had to be maintained in the delay interval). In contrast with the effect of lexical status (largest at the mid-line central site), the effect of number of items to be recalled was largest at the left frontal site. Figure 5 indicates that, starting at about 1000 msec after stimulus offset, the negativity over the left frontal site was largest in the word condition of the memory task (five items), next largest in the pseudo-word condition of the memory task (three items), and smallest in the control task (one item).

It might be argued that greater familiarity with the phonological structures of the words in comparison with the pseudo-words was responsible for the ERP results (Hulme, Roodenrys, Brown, & Mercer, 1995). Because the words consisted of familiar combinations of familiar syllables, while the pseudo-words consisted of unfamiliar combinations of familiar syllables, it is possible that familiarity may have affected ease of rehearsal. However, the phonological short-term memory studies reviewed above suggest that ERP indices of phonological rehearsal effects would most likely be manifested in the amplitude of the left frontal negativity, and not at the central sites where the lexicality effect was most pronounced.

The timing of the word/pseudo-word topographic differences suggested that the influence of lexical status upon retention began during encoding, starting with the presentation of the second item, and continued through the delay interval. Since the topographic differences occurred only in the recall task, and only after presentation of the first item, they were not likely to have been indices of the automatic activation of lexical codes postulated to occur as words are initially processed. Nor were the word/pseudo-word topographic differences likely to be only the remnants of lexical processing that occurred during intentional encoding for memory, since there was no such word/pseudo-word difference for the first item, and the difference was most pronounced and systematic during retention. Rather, the pattern of ERP activity suggests that the ERPs indexed the intentional maintenance of lexical codes subsequent to their activation, and that the maintenance process operated in parallel with encoding of later items in the stimulus series and then continued throughout the retention interval. Hence, the timing and topography of the ERPs in the Ruchkin et al. (1999) study support the contention that lexical processes contribute to verbal working memory maintenance operations when words are consciously held in working memory.

3.4. Lexical and semantic activation

Orthogonal variations of lexical and semantic properties of words to be remembered also influenced brain activation during retention, with different activation patterns for lexical and semantic manipulations (Ruchkin et al., unpublished data). A series of four different words was presented visually at a rate of one word/sec, followed by a 3500 msec delay interval that terminated with a serial recall test. All four words in a series had the same ombination of levels of frequency (high or low -- lexical variation) and concreteness (concrete or abstract -- semantic variation).

To delineate ERP activity that was specific to memory operations, there was a non-memory control task in which the stimulus and post-delay interval response processing demands were similar to those in the memory task, but there was no memory requirement in the delay interval. Subjects searched the series of four words for occasional deviant trials with a repeated word (probability = .10). Subjects were instructed to respond to a deviant trial with a finger movement immediately after presentation of the last word in the series, and to withhold the movement if the trial was not a deviant. At the end of the delay interval, an alphabetic character was displayed, and the subjects’ task was to say the letter in the alphabet that was in the third position after the letter in the post-delay interval display. For example, if the displayed letter was "j", then the response should be the letter "m". Only non-deviant control trials were used in the ERP analyses. The same words were used in the memory and control tasks, but with different combinations of four word series.

Error rates in the serial recall task were: 2.25% for high frequency, concrete words; 3.25% for high frequency abstract words; 5.05% for low frequency, concrete words; 8.16% for low frequency, abstract words. There were significant effects upon error rate of word frequency (F1,11 = 11.65, p = .0058) and concreteness (F1,11 = 8.55, p = .014). The error rate for the alphabet search that followed the delay interval in the non-memory control task was 1.13%. Debriefing indicated that only 7 of the 12 subjects in the study employed meanings of the words in their memorization strategy.

The ERPs for all subjects (n=12), regardless of memorization strategy, displayed a sustained increased negativity for low frequency words during the delay interval, with the effect of word frequency being maximal over mid-line central-parietal scalp (Fig. 6, upper panel, dashed lines). No such sustained difference was found in the control task (Fig. 6, lower panel, dashed lines). The average ERP amplitude over the last 2500 ms of the delay interval was used as a measure of ERP activity in the delay interval. ANOVA of these ERP amplitude measures at the six scalp sites in the centro-parietal mid-line region (C3 Cz C4 P3 Pz P4) that displayed the largest variation in sustained negativity as word frequency was manipulated revealed a significant effect of word frequency in the memory task (F1,11 = 11.23, p = .0065) and no significant effect in the control task (F1,11 = 0.63). Estimates of the temporal activation of the brain sources underlying the sustained increased negativity for low frequency words indicated that the sustained increased negativity started during presentation of the third word in the series.

The ERPs for those subjects (n=7) who used the meanings of the words in their memorization strategy displayed a sustained increased negativity for abstract words in the delay interval, with the effect being largest over frontal mid-line and left frontal-temporal scalp (Fig. 6, upper panel, solid lines). No such sustained difference was found in the delay interval of the control task (Fig. 6, lower panel, solid lines). ANOVAs of the ERP amplitude measures at the five scalp sites in the left frontal and frontal mid-line region (F7 F3 Fp1 Fz F4) that displayed the largest variation in sustained negativity as concreteness was manipulated revealed a significant effect of concreteness in the memory task (F1,6 = 10.81, p = .017) and no significant effect in the control task (F1,6 = .11). Estimates of the temporal activation of the brain sources underlying the sustained increased negativity for abstract words indicated that the sustained increase started during presentation of the second word in the series.

Concreteness also affected phasic ERP responses synchronized to the presentation of each word, with there being increased positivity in the ERP responses to abstract words in both the memory and control tasks. The scalp topography of these phasic responses to the stimuli differed from the sustained effect in the delay interval. In addition to the polarity difference, the phasic effect during stimulus presentation had a more posterior scalp distribution and was not lateralized to the left. Thus it is not likely that the same brain (and therefore cognitive) processes underlie the effects of concreteness upon the phasic ERP activity in the stimulus interval and the sustained ERP in the delay interval.

The effect of word frequency upon sustained ERP activity in the delay interval is not likely to be due to pre-lexical processing such as phonological rehearsal. Other studies have reported increased sustained negativity over left frontal scalp as phonological load increasd (Ruchkin et al., 1990; Ruchkin et al., 1992; Ruchkin et al., 1994; Ruchkin et al., 1997a). In contrast, the shift from high to low frequency words, which caused a significant increase in error rate, resulted in an increased sustained negativity over bilateral centro-posterior scalp. The topography associated with the word frequency effect is congruent with the effect of the word versus pseudo-word manipulation (see above, (Ruchkin et al., 1999)). However, the variation in sustained negativity as concreteness was manipulated only may be an indirect ERP sign of semantic processing during retention of verbal material, since its topography is consistent with findings from short-term memory studies where phonological load or general attentional demands were manipulated (Ruchkin et al., 1990; Ruchkin et al., 1992; Ruchkin et al., 1994). The error rates suggest that abstract words demand greater cognitive resources in short-term memory than concrete words. In such a case, other processing systems may have become more active as a compensatory effect when abstract words had to be retained. Nevertheless, the finding that variation of concreteness affected brain activity during the delay interval does indicate that semantic processes actively support the maintenance of verbal information in short-term memory.

Figure 6. Across-subjects averaged scalp difference ERPs in a verbal shortterm memory task (serial recall of a list of four words - upper panel) and a non-memory control task (detect a repeated item in the list of words – lower panel). There was no memory requirement in the post-stimulus delay interval of the control task. Difference ERPs show the effects of concreteness and word frequency upon brain activity in the stimulus and delay intervals. The word frequency effect was revealed by subtracting ERPs elicited by lists of high frequency words from ERPs elicited by low frequency words (Low - High), pooled over abstract and concrete words and averaged over all subjects (n=12). The concreteness effect was revealed by subtracting ERPs elicited by lists of concrete words from ERPs elicited by abstract words (Abstract - Concrete), pooled over high and low frequencies, and averaged across those subjects (n=7) who employed word meaning in their memory strategy. The time axis extends from 360 ms before to 7520 ms after stimulus onset. Stimulus duration was 4000 ms and the duration of the subsequent delay interval was 3520 ms.

Note that in the memory task, during the post-stimulus retention interval, the concreteness effect is largest over frontal scalp, with greater negativity for abstract words. In contrast, the frequency effect is largest over centro-posterior scalp during retention, with greater negativity for low frequency words. In the non-memory control task, the effects of concreteness and frequency are relatively small in the post-stimulus interval. These results indicate that semantic codes also contribute to the maintenance of verbal information in short-term memory, and the combination of brain sources associated with the semantic processes differs from the combination of sources associated with the lexical processes.

3.5. Summary: Verbal working memory in serial recall and match/mismatch tasks

Fig. 7 summarizes in schematic form the timing results obtained from our ERP studies of verbal working memory in delayed serial recall and match/mismatch paradigms. These data support the view that multiple, simultaneously active processes are involved in the short-term maintenance of verbal information (Cowan, 1988; Cowan, 1995; Cowan & Kail, 1996). The data are consistent with the notion that phonological and lexical-semantic codes may interact throughout the retention interval, as opposed to only during retrieval. The activated lexical-semantic codes continuously counteract degradation of the material in the phonological buffer, while phonological codes may have a role in counteracting degradation of the lexical semantic codes (Martin & Romani, 1994; Martin & Saffran, 1997). The timing of the ERPs indicates that the sustained lexical-semantic processes start during encoding, evidently as information concerning the properties of the material to be memorized builds up, and continues in the subsequent retention interval. Along with behavioral data (Penney, 1989), the ERP time-courses also suggest that the modality-specific auditory-verbal store is more durable than the modality-specific visual-verbal store.

Figure 7. Schematic of the timing of hypothesized stores that contribute to the operations of verbal working memory.

3.6. Activation of long-term memory and retention

Cameron et al. (2002) used ERPs to test the premise that retention of words in short-term memory involves activation of the words’ semantic representations in long-term memory (i.e., activated long-term memory provides a representational basis for short-term maintenance of information). Cameron et al. conceptualized the instantiation of the activation process as being akin to temporally extended priming. As words are initially processed, their various representations in long-term memory are activated. The depth of processing of these words determines the levels of activation (e.g., phonological, semantic). If no conscious effort is exerted to maintain the words in working memory, then the activation of their representations decays. If there is a conscious effort to maintain the words in working memory, then the level of activation of their long-term representations remains relatively high during the retention interval.

Cameron et al.’s (2002) approach was to contrast the degree of activation in the semantic neighborhood of a series of three words used in two tasks; one that required retention of the meanings of the words and one that did not require retention of the words. The three words in the stimulus series were associated with each other. The degree of semantic activation was determined from the ERP response to an incidental probe word presented during a delay interval that followed the stimulus series. The probe word was either unrelated to the last word in the stimulus series or semantically and categorically related to the last word in the series. In effect, the three words in the stimulus series were primes and the probe word was the target for the primes. Both the series of priming words and the probe word were presented aurally. Participants were instructed that the probe word was a distractor stimulus to be ignored, and no response was to be made to it.

The degree of semantic activation was assessed by means of a ERP phenomenon elicited by the probe, referred to as N400, that is sensitive to the extent to which semantic representations of the eliciting word have been activated by the previous presentation of verbal material (Bentin, McCarthy, & Wood, 1985; Bentin, Kutas, & Hillyard, 1993; Holcomb, 1988; Nobre & McCarthy, 1994; Bentin, McCarthy, & Wood, 1984). N400 is a phasic negative deflection that is most prominent 200-800 ms after stimulus onset for auditory stimuli (Bentin et al., 1993; Holcomb & Neville, 1990). N400 negativity is reduced when the eliciting word has been primed by (i.e., is in the semantic neighborhood of) previously presented verbal material and is increased when the eliciting word has not been primed (Kutas & Hillyard, 1989; Bentin et al., 1993; Holcomb, 1988). This variation in N400 negativity can occur even when participants are unaware of the semantic relationships (Bentin et al., 1984), and when elicited by an unattended stimulus, provided that the previously presented material was attended (Kellenbach & Michie, 1996).

Kutas and Federmeier (2000) have argued that N400 amplitude reflects the degree of difficulty with which verbal material is accessed in long-term memory. The degree of access difficulty (or ease) depends upon the extent to which the verbal material is incompatible (or compatible) with the context established by previously presented verbal material. Thus, Cameron et al. (2002) reasoned that the access difficulty (or ease) would depend, to some extent, on the degree to which the semantic representations of previously presented material were activated. That is, increased activation of the material’s semantic representations would establish greater contextual constraints on accessing the long-term memory representations of subsequent verbal material. Hence, if maintaining the three priming words in working memory involved enhanced activation of their semantic codes, then the N400 elicited by a related probe would be smaller, and the N400 elicited by an unrelated probe would be larger, in the Memory task than in the Control task.

In the Memory task, a set of three priming words were presented aurally at a rate of one word per second, followed by a 4000 ms delay interval. At the end of the delay interval, a word was displayed until the participant verbally indicated whether the word matched or did not match the meaning of any of the three primes. Matches occurred on 50% of the memory task trials. An incidental probe word was presented aurally 2000 ms after onset of the delay interval. On 50% of the trials, the probe word was semantically and categorically related to the last word of the previously presented priming set. On the other trials, the probe was neither semantically nor categorically related to any of the primes.

The Control task was designed such that 1) there was no memory requirement during the 4000 ms delay interval, 2) the depth of processing of the three priming words preceding the delay interval was comparable to that in the Memory task, and 3) the operation at the end of the delay interval was comparable in difficulty to that in the Memory task. To eliminate the memory requirement, there was no contingency between the operations preceding and following the delay interval. Before the delay interval, participants decided whether the priming set contained one word that was unrelated (not associated) with the other two words. Participants were instructed to respond immediately with a vocal response if they detected an unrelated word in the priming set, and not to respond when the words in the priming set were all related. On 90% of the trials the three primes were associated (and hence there was no vocal response), and only these trials were used in the analysis of the ERP data. In Control trials, the task at the end of the delay interval involved adding a visually presented pair of two-digit numbers. The display of the pair of numbers terminated when the participant responded verbally with the sum. As in the memory task, an incidental probe word was aurally presented 2000 ms after onset of the delay interval, and was semantically and categorically related to the last word in the priming set on 50% of the trials and unrelated with any of the words in the priming set on the remaining trials.

Figure 8. Across-subjects (n=24) averaged ERPs at posterior scalp sites, where the effects of probe status and task were largest, for the four combinations of incidental probe status and task. The vertical level of the time axis for each recording corresponds to the average amplitude in the 100 ms interval preceding probe onset. The time base extends from 100 ms prior to the incidental probe onset to 1000 ms after probe onset.

Note that the N400 activity elicited by unrelated probes was largest in the memory task compared to the control task, and that the N400 activity elicited by related probes was smallest in the memory task compared to the control task. This combination of results indicates that there was greater activation of semantic representations in the delay interval of the memory task than in the non-memory control task.

The ERP responses to the probe for each of the four combinations of task (memory, control) and probe status (related or unrelated to the preceding three words) are presented in Fig. 8. ERPs elicited by unrelated probes in both memory and control tasks displayed enhanced N400 activity over posterior scalp compared with ERPs elicited by related probes. Fig. 9 displays the contrast between unrelated-minus-related probe difference waveforms for the memory and control tasks. Effects specific to the memory and control tasks, other than that of probe status, were approximately canceled in these difference waveforms. Fig. 9 makes clear that the effect of semantic relatedness upon N400 activity was greater in the memory task. Thus it can be inferred from Fig. 9 that retention of previously presented words results in a generally enhanced level of semantic activation during the retention interval, as indexed by N400 amplitude and duration.

Figure 9. Across-subjects average difference ERPs for the unrelated minus related incidental probe subtraction. This figure depicts the effect of the interaction between task (memory, non-memory control) and probe status (related, unrelated) upon N400 activity elicited by the incidental probe. Note that the effect of whether the incidental probe word is semantically related or unrelated to the three priming words preceding the delay interval is greater in the memory task.

The question of whether there is specifically greater activation in the semantic neighborhood of words when they are maintained in short-term memory is addressed by comparing the memory and control task ERP waveforms for related probes (red waveforms in Fig. 8). Note that, for the related probes, the N400 activity in the memory task has a shorter duration and somewhat lower amplitude than in the non-memory control task. This effect is most marked at posterior temporal and occipital sites. Thus, it can be inferred from the smaller N400s elicited by the incidental probe in the memory task that activation in the semantic neighborhood of previously presented words is greater when the words are consciously maintained in short-term memory.

These results show that retention of verbal material in short-term memory tasks that emphasize meaning is accompanied by extended, enhanced activation of the material’s semantic representations. The effect of semantic relatedness upon the incidental probe is not readily explained by verbal short-term memory being a separate buffer into which copies of long-term memory representations are transferred. A more parsimonious explanation is that verbal short-term memory is a process that involves continuous maintenance of long-term memory representations at enhanced levels of activation. Thus, the long-term memory representations of the related probe words may be easier to access in the memory task than in non-memory control task (Kutas & Federmeier, 2000). Conversely, the long-term memory representations of the unrelated probe words may be more difficult to access in the memory than non-memory control task due to enhanced activation of nonmatching features by the material held in short-term memory.

3.7. Sentence processing and semantic relatedness

The verbal working memory experiments reviewed above employed lists of unconnected items. Such paradigms have provided a useful but limited view of brain activity involved in typical verbal working memory operations. The study described below extended this view by using a sentence, rather than a series of unconnected words, as the stimulus, and requiring retention of the meaning of the sentence.

Comprehending a sentence involves a process of semantic and syntactic binding (Hagoort, 2000), whereby the meanings of the words in a sentence are related to one another and maintained in short-term memory as part of an integrated overall representation. There is evidence for a postinterpretative process that maintains thematic role relations following their syntactic computation (Caplan & Waters, 1999). This process is more error prone and/or engenders more brain activation when a sentence expresses more propositions (Caplan, Alpert, & Waters, 1998) suggesting that propositions and the thematic role relations they express are maintained by a capacity-limited semantic short-term memory process. Haarmann et al. (2002) sought neurophysiological evidence for such a process by manipulating semantic relationships within a sentence and analyzing the effects of the manipulation upon ERP and EEG activity in a post-sentence retention interval. The ERP results further provided support for Crowder’s (1993) proceduralist view of memory, namely that brain systems that process particular items of information also subserve storage of those items. Finally, the EEG results provided information on the interactions among brain systems involved in the initial rocessing and subsequent retention of the sentence.

The short-term memory process evidently depends in part upon interactions between frontal and posterior cortex implemented by the operation of frontal-posterior projection loops. The loops projecting from frontal cortex mediate the focusing of attention upon representations in posterior cortex to be retained, while the projections from posterior cortex provide information about the state of posterior cortical systems to frontal neural networks. Presumably the resulting influence of frontal and posterior cortical systems upon each other is actualized by an increase in synchrony between neural circuits in the two brain regions. Support for this view is provided by studies of verbal and visual-spatial short-term memory tasks in which the synchronization between EEG recordings from different scalp sites was analyzed with EEG coherence measures. The patterns of coherence between EEG recordings from frontal and posterior sites were found to differ markedly between the stimulus presentation and subsequent retention intervals (Sarnthein, Petsche, Rappelsberger, Shaw, & von Stein, 1998; von Stein & Sarnthein, 2000; Engel & Singer, 2001). These findings, when combined with evidence that the same brain regions are active during both the initial processing and subsequent post-stimulus retention, support the position that short-memory operations utilize specific patterns of connectivity among brain regions, and not buffers that are specialized to short-term memory storage.

This position was tested by Haarmann et al., (submitted) in a study of ERP activity during and following the reading of filler-gap sentences. Sentences, consisting of six phrases, were presented visually on a phrase-byphrase basis over a 4500 ms interval, followed by a 2000 ms delay inte