Seeing Ɔ, remembering C: Illusions in short-term memory

Marte Otten; Anil K. Seth; Yair Pinto

doi:10.1371/journal.pone.0283257

Abstract

Perception can be shaped by our expectations, which can lead to perceptual illusions. Similarly, long-term memories can be shaped to fit our expectations, which can generate false memories. However, it is generally assumed that short-term memory for percepts formed just 1 or 2 seconds ago accurately represents the percepts as they were at the time of perception. Here 4 experiments consistently show that within this timeframe, participants go from reliably reporting what was there (perceptual inference accurately reflecting the bottom-up input), to erroneously but with high confidence reporting what they expected to be there (memory report strongly influenced by top-down expectations). Together, these experiments show that expectations can reshape perceptual representations over short time scales, leading to what we refer to as short-term memory (STM) illusions. These illusions appeared when participants saw a memory display which contained real and pseudo-letters (i.e. mirrored letters). Within seconds after the memory display disappeared, high confidence memory errors increased substantially. This increase in errors over time indicates that the high confidence errors do not (purely) result from incorrect perceptual encoding of the memory display. Moreover, high confidence errors occurred mainly for pseudo-to-real letter memories, and much less often for real-to-pseudo-letter memories, indicating that visual similarity is not the primary cause of this memory-bias. Instead ‘world knowledge’ (e.g., which orientation letters usually have) appear to drive these STM illusions. Our findings support a predictive processing view of the formation and maintenance of memory in which all memory stages, including STM, involve integration of bottom-up memory input with top-down predictions, such that prior expectations can shape memory traces.

Citation: Otten M, Seth AK, Pinto Y (2023) Seeing Ɔ, remembering C: Illusions in short-term memory. PLoS ONE 18(4): e0283257. https://doi.org/10.1371/journal.pone.0283257

Editor: David Keisuke Sewell, The University of Queensland, AUSTRALIA

Received: August 24, 2022; Accepted: March 5, 2023; Published: April 5, 2023

Copyright: © 2023 Otten et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data is publicly available from the OSF database (https://osf.io/dfbva/).

Funding: AKS is grateful to the European Research Council (ERC-2020-ADG, grant 1010192654) for support.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Memory is an indispensable part of human cognitive functioning. If we see a new colleague again after a two week vacation, we can usually rely on long term memory (LTM) to put the correct name to the face. If we turn our heads away from a scene, we can rely on our short term memory (STM) to recollect what we just saw. STM is an umbrella term for memory that stores information for a short period of time (seconds to minutes). Different types of STM have different capacity and time-limits. STM is often hypothesized to consist of short-lived (<500 ms) iconic memory [1] and longer lasting working memory [2]. Sligte and colleagues have additionally suggested the presence of an intermediate form of STM, namely fragile memory [3], which, like WM, lasts at least 4 seconds, but unlike WM has a capacity of 5–15 items. Here, we present 4 studies that explore whether STM in general is susceptible to false memories just like LTM has been shown to be [4, 5]. These experiments specifically focus on a subset of false memories, namely what we call “memory illusions”: cases were people report with high confidence to have seen something that was not presented to them, i.e., the illusory experience of a reliable and reportable memory. An example of this would be clearly remembering that you put your keys on the dining room table, while in reality you left them on the side table by the front door. Focusing on memory illusions instead of general false memory reports allows a separation of such incorrect but seemingly true memories from memory lapses that are filled in by best guesses (such as when you have no clue where you put your keys, but you check out the dining room table, because you very often drop them there). In such memory lapses, the reliance on pre-existing knowledge is a useful strategy to consciously fill gaps in memory, but it does not reveal whether prior knowledge can actually shape the content of STM. Here, we propose that memories in short term memory are at least partially shaped by world knowledge: prior expectations about the likely events or objects, based on lifelong learning and development. In this case, the relevant expectations relate to a highly overlearned category of visual symbols, namely letters of the alphabet.

Recollections of visual stimuli from STM often serve as a proxy for perception [6]. This illustrates the general assumption that STM is a highly reliable storage of recent events: failures to correctly report information that was only recently presented is seldom interpreted as a memory error, but usually seen as a perceptual error. For example, De Gardelle and colleagues [7] showed participants displays which sometimes contained mirrored letters (i.e., a pseudo-letter such as Ɔ). The participants, however, often indicated seeing the real counterpart of the rotated letter (i.e., they report seeing C). De Gardelle and colleagues (2009) [7] conclude that this reflects a perceptual illusion—in other words that their participants make an incorrect perceptual inference at the time of perception. They did not explore the possibility that their results could be due to memory errors instead of perceptual errors.

However, recall of perceptual information is known to be subject to errors over many time scales. Research in LTM has shown that contextual information that is present both during [8, 9] and after [10, 11] an event can shape the memory for that event, when probed again after a prolonged interval. Arguably, these LTM illusory memories are the result of an adaptive process, in which memory is an efficient, (re-)constructive system which integrates world knowledge within long-term memories to recreate past observations [4, 5]. Indeed, some studies show that world knowledge alone (such as knowledge about social roles) is sufficient to generate memory errors. For example, Kleider and colleagues [12] showed participants faces combined with roles (professor, drug dealer, artist). After a 20-minute delay, participants were more likely to incorrectly attribute criminal labels to faces with stereotypically black features. This suggests that an internal bias associating black faces with crime can shape memory. Similar effects were shown for male and female names that were linked to gender-(in)consistent occupations [13]: illusory recollections were more likely to follow gender-based expectations. These results suggest that both external, contextual information and internal expectations about the world can change the content of LTM, resulting in illusory memories.

Here, we asked whether prior knowledge can influence not just LTM, but also STM, resulting in the generation of illusory short term memories. There is some suggestive evidence that illusory memories may indeed occur in STM: Just 3–4 seconds after reading a list of 3 or 4 interrelated words, participants were more likely to confirm that they saw a semantically related word that was not in the original list, compared to an unrelated new word [14–16]. Interestingly, world knowledge alone may underlie illusory memories in STM. For example, one’s cultural experience with music can induce erroneous responses about musical mode and tonality just 1 second after hearing the sequence of tones [17, 18].

Though suggestive, these studies are not yet compelling proof for illusory memories arising in STM, based on knowledge about the world. First, these previous studies did not utilize designs able to distinguish perceptual errors from memory errors. Specifically, if you only test at one single moment after disappearance of the stimuli, then any illusion can either be perceptual (the illusion arose while the participants perceived the stimuli) or memory-based (the illusion arose after perception was finished). One way to resolve this ambiguity is to test participants at (at least) two different moments after stimulus offset. If the frequency of reported illusions increases over time, then at least part of the illusions can be attributed to memory processing rather than perceptual encoding. After all, at both time points initial perceptual processing was finished, so if the illusions arose completely during perception, they ought not to increase over time. In short, characterising the temporal evolution of illusions provides one means of ascertaining whether an illusion is primarily perception-based or memory-based.

Returning to the results of De Gardelle et al. [7], these results show that people report memories of real letters when pseudo-letters were shown. These reports could be the result of illusory memories in which pseudo-letters were progressively replaced with more expected real letters. However, De Gardelle and colleagues interpret their results as perceptual illusions, not memory illusions. Contrarily, other results that are interpreted as STM illusions [7, 14–19] could equally be the result of perceptual illusions that arise when the memory is encoded, after which it is stored and maintained, and subsequently reported, or even just reflect a best guess.

Illusory memories should result in high confidence erroneous responses, where the participants has access to a memory trace (albeit an incorrect memory trace). In the absence of confidence ratings, the erroneous responses could actually consist of low confidence responses or even outright guesses. In these cases, the response is most likely not the result of an illusory memory. Instead, it is more likely to result from the absence of a reliable memory, which is compensated by conscious inferences about what is likely to have occurred. Therefore, isolating high confidence responses is necessary to identify those responses that reflect a participant having an actual, albeit false, memory of perceiving something that was not there.

Here we report 4 experiments that test whether STM is susceptible to illusory memories. In all experiments participants see memory displays that contain both real letters, with the standard orientation, and pseudo-letters, which were mirrored. We investigate the occurrence of high-confidence errors (in which participants report seeing a real letter when a pseudo-letter was shown, or vice versa) at different time-points after the display has disappeared. These experiments test whether STM is susceptible to illusory memories, even when the visual information has only just disappeared, and the perceptual memory trace should still be intact [20].

To rule out that these errors reflect a perceptual error that has directly been encoded into STM, we test whether illusory responses increase over time: If the illusions are simply the result of misperceptions, their frequency of occurrence should remain the same over the time-course of STM—since in all cases initial perceptual inference has been completed. Therefore, if the number of reported illusions increases as the retention interval increases, this is best explained by memory-related processes taking place during the retention interval, so during STM storage itself, and not in the perceptual or encoding process. Moreover, we exclude the possibility that the errors are a result of guessing in the absence of reliable memories / reliable perception by focusing on high confidence errors: errors where the participant explicitly indicates to know the answer, and has not just guessed.

All experiments rely on the extensive experience participants have with the alphabet. Previous experience with letters should have provided participants with abundant evidence that letters typically appear in their standard form. This should provide strong internal expectations about the appearance of the letters of the alphabet. Therefore, occasional instances where the mirrored pseudo-letters are presented might lead observers to incorporate these internal expectations within their memory, and generate illusory memories of true letters, where in fact a pseudo-letter has been shown.

In 4 experiments we thus test 3 hypotheses: 1) Illusions occur in STM: high confidence reports of having seen a real letter when a pseudo-letter was shown, or vice versa, occur within 1–2 seconds after a memory display has disappeared. 2) These reports reflect illusory memories, and are not the result of incorrect perceptual encoding: the high confidence illusory reports become more prevalent as memory deteriorates, through the passage of time [21] or through interference [22], and 3) The illusory memories are driven by world knowledge: high confidence illusory reports are more likely to involve pseudo-letter to real-letter errors (as real letters are well-studied and much more prevalent in the real world) than the other way around.

Experiment 1

Methods

Training.

Prior to the main experiment, participants were trained on a cued change detection task. In this training task, participants were instructed to indicate whether the rectangle at a cued location in a memory display had the same orientation as the rectangle at that same location in a test display. The memory and test displays each consisted of eight white rectangles (1.16° × 0.29° in size) which had horizontal, vertical or oblique (45° or 135°) orientations. The orientations were pseudo-randomly selected, such that each orientation was equally likely, and that each orientation appeared at least 1, and at most 3 times. The rectangles were presented on a black background, around a red fixation dot (radius: 0.4). Individual rectangles were evenly distributed on an imaginary circle (radius 4.68°) around fixation. The cue (indicating the target location) consisted of a yellow line that was at one end close (±0.93°) to fixation and at the other end close to the centre of one rectangle (± 1.6°). The experiment was programmed in Matlab version 2012b using the Psychophysics Toolbox routines [23]. On each trial the memory display was presented for 0.25 seconds. This was followed by a blank screen of 0.1–1 seconds, a cue for 0.5 seconds and a test display, which remained visible until response. The memory and test display were identical, except that the orientation of the rectangle at the cued location differed 90 degrees between both displays on a randomly selected 50% of the trials. The cue appeared either 100 milliseconds, or 1 second after offset of the memory display. In some trials the cue appeared during a blank interval in between memory display and test display, in other trials the cue appeared together with the test display. In all trials the cue remained visible for 0.5 seconds, and the test display remained visible until response.

Participants could only participate in the actual experiment, if they scored higher than 75% correct (averaged across all trials) on the final 10 blocks of training. We adopted this procedure to ensure that all participants were motivated, and possessed a minimal proficiency at visual memory tasks. See Pinto, Sligte, Shapiro, & Lamme [24] and Sligte, Scholte, & Lamme [3] for a similar training & exclusion procedure. Since only 10% of participants were excluded from participating in the main experiment based on the training scores, the training procedure was not part of Experiments 2, 3 and 4 in order to not make the overall time on task too long.

Participants.

45 participants started training, 5 participants were excluded based on their training results. The remaining 40 participants (28 female; age range 19–30 years, average age 22.2 years) all had normal or corrected-to-normal vision. All participants were naïve to the purpose of this experiment. All participants gave their written informed consent to participate in the study, which was approved by the local ethics review board of the University of Amsterdam. The participants participated for monetary compensation (10 euro per hour) or student credits.

Stimuli.

The memory and cue displays were set up in the same way as the displays in the training session, with 6 or 8 letters replacing the tilted bars. Pseudo-letter were mirrored (i.e. flipped from left to right). Therefore only letters that changed when mirrored were included in the experiment (i.e. the symmetrical letters A,H,I,M,O,T,U,V,W,X,Y were excluded from the memory and response displays). When the memory display consisted of 8 letters, it contained 2 pseudo-letters. When it contained 6 letters, it had a 50% chance of containing 1 pseudo-letter and a 50% chance of containing 2 pseudo-letters.

To increase memory decay through task-irrelevant interference, a second display was included in each trial. This display consisted of randomly selected letters, unrelated to the memory display. Furthermore, it contained the same proportion of real and pseudo-letters as the memory display. Participants were instructed to ignore this second display.

The response display consisted of 6 letters: the target letter (with the same orientation as in the memory display, or mirror reversed), two other randomly selected letters from the memory display, and 3 letters that did not appear in the memory display.

The target letter was a real letter in 75% of the trials and a pseudo-letter in the remaining 25%. If the target was a pseudo-letter, then it appeared as a response option randomly on half of those trials while on the other half of pseudo-letter trials only the real counterpart was one of the response options. If the target was a real letter, then it appeared as a response option on a randomly selected 83% of the trials, while its mirror reversed (pseudo-letter) version appeared as a response option on the other 16% of the trials. This ensured that of the critical trials (trials where participants could exhibit memory illusion: trials in which a real- or pseudo-letter target was combined with its mirrored counterpart in the response screen) about half were real-to-pseudo trials (approximately 13% of all trials), while the other half were pseudo-to-real trials (again, approximately 13% of all trials). Together, the critical trials made up ~220 trials, 26% of all trials in the experiment. It is important to note, though, that throughout the experiment participants were more likely to encounter real letters as the target, and as the correct response option, than pseudo-letters, in line with the overall expectation that real letters are the standard letter-form.

Procedure.

The experiment consisted of 2 sessions of 18 blocks of 24 trials, see Fig 1. The first session started with a practice block of 12 trials (this block was not included in the analysis). The experiment had three memory conditions. In two conditions the memory probe appeared 0.75 seconds after offset of the memory display, in the third memory condition, the probe appeared 3 seconds after offset of the memory display. If the probe appeared 0.75 seconds after memory display offset, it either appeared together with the irrelevant second display (the Short-with-interference condition), or it appeared in isolation (the Short condition). If the probe was presented 3 seconds after offset of the memory display (the Long condition), it appeared in isolation. If the memory probe appeared in isolation (the Short and Long conditions), the irrelevant second display appeared 5 seconds after memory display offset, which meant that the interval between the probe and the irrelevant second display was either presented for 1.5s (Long delay) or 3.75s (short delay). In all cases, the memory probe was shown for 0.5 seconds and the irrelevant second display was presented for 0.5 seconds. The three trial-types were randomly intermixed.

Download:

Fig 1. Experiment 1—Procedure.

The memory display consisted of 6 or 8 letters. It contained 1 to 2 pseudo letters (set size 6) or 2 pseudo letters (set size 8). In all 3 memory delay conditions the trial started with the memory display appearing for 0.25 seconds. Subsequently in the Short and Long conditions (panel A) the memory cue appeared after a blank of 0.75 seconds (Short) or 3 seconds (Long), for 0.5 seconds, on a blank screen. Subsequently, 5 seconds after offset of the memory display, an irrelevant second letter display, containing the same amount of (randomly selected) real and pseudo-letters as the memory display, appeared for 0.5 seconds. The Short with Interference condition (panel B) was the same as the Short condition, except that now the second letter display appeared together with the cue. In all conditions, after offset of the second letter display, the participant first indicated the identity of the letter at the cued location in the memory display. Subsequently she indicated confidence in her judgment (1 = guess, 4 = certainty).

https://doi.org/10.1371/journal.pone.0283257.g001

Participants were instructed to report the letter that appeared at the cued location. They were explicitly instructed not to confuse real and pseudo-letters and received feedback on their performance after each trial. They were instructed to select "none of these" if the target letter was not a response option. After selecting the target letter, participants indicated their confidence by clicking on one of four boxes (from 1–4, 1 = guess, 4 = certain).

Analysis.

For each memory delay (short, short with interference and long) and each target type (pseudo- and real letter targets) the occurrence of illusory errors (real-to-pseudo or pseudo-to-real errors) and other errors (selection of one of the unrelated distractor items) was calculated.

Exclusion of participants with too few high confidence critical trials in one or more cells of the design (10 trials or less) were excluded from the analysis resulted in an inclusion of 23 participants (17 participants were excluded) when the results were averaged across the set size conditions. Included participants had on average 53.8% high confidence trials over all critical conditions (range: 36.3–93.2, sd: 13.9). Excluded participants had on average 26.2% high confidence trials over all conditions (range 3.6–47.7%, sd: 13.0).

The resulting data were analysed in a Bayesian repeated measures ANOVA in JASP 0.16.4 [25], with the factors Target Type (real or pseudo-letter), Memory Delay (short, short with interference and long) and Error Type. Within this analysis, the posterior odds of all different combinations the three factors and their interactions are compared to the prior explanatory value of each model. Based on this analysis, we selected the model with the best fit, reporting the BFm. Individual model components (main effects and interactions) were additionally evaluated based on their BFincl across all models. Bayes Factors were interpreted as follows: BF < 0.1: strong evidence for the null hypothesis. 0.1–0.33: substantial evidence for the null hypothesis. 0.33–1: anecdotal evidence for the null hypothesis. 1–3: anecdotal evidence for the alternative hypothesis, 3–10: substantial evidence for the alternative hypothesis, >10: strong evidence for the alternative hypothesis [26].

We investigated our three main hypotheses: hypothesis 1–do illusions occur in STM?—by determining if participants confused pseudo-letter targets with their real counterpart with high confidence, or the other way around, more often than other high confidence errors (main effect of Error Type). We tested our second hypothesis—do illusions increase as memory deteriorates?—by measuring if illusions increased from the Short to the other memory conditions (interaction between Memory Delay and Error Type). Finally, we tested our third hypothesis—does world knowledge drive STM illusions?—by investigating if high confidence errors confusing pseudo-letter targets for their real counterparts occurred more often than the other way around (Target Type interacts with Error Type).

Results

Fig 2 shows the illusory and other error rates in high confidence trials. The Bayesian repeated measures ANOVA showed that a model with main effects for Error Type, Target Type, and Memory Delay, and a Target Type * Error Type and Memory Delay * Error Type interaction had the best fit (BFm = 18, see S1 Appendix for full model comparison and Table 1 for an overview of BFincl for each model component).

Download:

Fig 2. Experiment 1.

Error rates in trials where participants indicated high confidence in their response. Percentages of expectation-induced errors and other errors are depicted for pseudo-letter targets (right display) or a real letter targets (left display). All error rates are reported for each of the three Memory Delay conditions.

https://doi.org/10.1371/journal.pone.0283257.g002

Download:

Table 1. Error rates ANOVA.

https://doi.org/10.1371/journal.pone.0283257.t001

Hypothesis 1 was supported: Illusions (pseudo-to-real or real-to-pseudo) were much more prevalent than other errors (a main effect of Error Type, BFincl>1000). The interaction between Error Type and Memory Delay, (BFincl = 4.21) shows support for hypothesis 2: as memory deteriorates with a larger memory delay or more visual interference, the increase in illusory errors (7% increase from 19% in the short delay to 26% averaged over the short with interference and long delay) is larger than other errors (4% increase, from 8 to 12%). The interaction between Target Type and Error Type (BFincl >1000) provides evidence for Hypothesis 3: when the target is a pseudo-letter, illusions are very common (37%) but other errors occur much less often (11%). For real letter targets, the illusory error rate is the same as the rate of erroneously selecting any other item from the response display (11% in both cases). Finally, there is only anecdotal evidence for a three-way interaction between Target Type, Error Type and Memory Delay (BFincl = 2.45): For pseudo-letter targets, illusory errors increase 12% as memory deteriorates through interference of time or visual information (29% illusions to 41% illusions). None of the other conditions are as sensitive for memory decay, with other errors for pseudo-letter targets increasing 3%, and illusory errors and other errors for real letter targets increasing 2% and 7%, respectively.

Discussion Experiment 1

In Experiment 1 we observed an abundance of pseudo-to-real letter memory illusions. In fact, when the probe was after an interfering visual display, high confidence pseudo-to-real illusions (43%) were almost as prevalent as high confidence selections of the actual pseudo-target (50%). That is, when participants were sure of their memory, the recollection was almost as likely to be illusory as it was to be real.

What causes these illusory reports? One option is incorrect perception or encoding when the memory is formed. However, Experiment 1 shows that expectation-induced memory illusions increase as memory declines. From the Short (memory probe 0.75 seconds after offset of the memory display with no visual interference) to the Long condition (memory probe after 3 seconds) or the Short with interference (memory probe after .75 seconds following an interfering display) the prevalence of high confidence expectation induced illusions increased from 29% to 39% and 43%, respectively. This indicates that high confidence errors are not purely the result of errors in perception or encoding: when probed 0.75 seconds after the memory display has disappeared, people correctly report what they saw in 70% of high confidence trials, thus showing that initial perceptual encoding was relatively intact. When the memory trace deteriorates, high confidence reports are moving towards an equal division between accurate and illusory memories. This effect must be due to processes ongoing in STM: Had these illusory memories been the result of changes in perception, they should have been present at the earlier probe as well.

Another cause of illusory reports could be confusion between the to-be-remembered target and the mirrored counterpart that is presented in the response screen. However, Experiment 1 shows that pseudo-to-real illusions were much more prevalent than real-to-pseudo illusions. This suggests that world knowledge is a more important driver of illusory memories than visual similarity, as visual similarity between to-be-remembered target and the mirrored counterpart that is presented in the response screen is clearly identical in both cases.

Yet another reason why illusory reports arise could be a response bias: Participant might just be more inclined to select real letters instead of pseudo-letters, since real letters are a well-known category. If the illusory reports are actually not illusions, but due to response bias, then the pattern of responses that can be attributed to a response bias (more expectation-induced illusions [Ɔ to C] than similarity-induced illusions [C to Ɔ]) should be present not only for the high-confidence responses; indeed, it should be even stronger for the low-confidence responses. This is because a response bias is present whether the participant thinks they remembered correctly or not, and a participant ought to rely more heavily on a response bias when they have only a weak memory of what they saw. However, in Experiment 1 the tendency to report more expectation-induced illusions than similarity induced illusions is clearly present in high confidence trials, but is much less pronounced in low confidence trials—an opposite finding to what one would expect if these effects were the result of a response bias: On high confidence trials pseudo-letter targets were more often confused for their real counterparts (36%) than on low confidence trials (22%). However, the reverse was observed when real letter targets were confused for their pseudo counterpart (high confidence: 11%, low confidence: 17%).

This also shows that the current illusory memories are quite different from earlier reports of swap errors in STM [27–30], where participants incorrectly report having seen features of a distractor, instead of the target feature. While the currently observed memory illusions are more likely to occur in high confidence trials than on low confidence trials, swap errors generally occur on low confidence trials instead of high confidence trials [28].

Experiment 2

The illusory memories observed in Experiment 1 seem to be driven by world knowledge, i.e. long-standing knowledge about the usual orientation of letters. In Experiment 2, we tested whether transiently induced expectations can also cause illusory memories in STM. To this end, we associated the color of the fixation cross with either a high or a low probability of a pseudo-letter target. If transiently induced expectations can induce illusory memories, then pseudo-to-real illusions should be more prevalent when the fixation color induced the expectation of a real letter target, than when it induces the expectation of a pseudo-letter target.

Experiment 1 also had different proportions of trials where the real letter counterpart of the target was presented in the probe display, compared to the pseudo-letter counterpart. Although the results of Experiment 1 (with its marked increase in illusory memories when memory decays) suggest that this difference in occurrence of the mirrored (pseudo) counterpart in the response screen over the entire experiment is not the cause of the memory illusions we observe, it is important to completely rule this out. Therefore, the response screen in Experiment 2 always included both the target and its mirrored counterpart. This equates the presence of real letter and pseudo-letter probes in the response display over the entire experiment, and makes sure that the participant cannot become biased to select one of the two options simply because it is generally more prevalent in the response screen.