Every year, a small group of working memory researchers meets in the idyllic Yorkshire Dales for a laid-back and friendly discussion meeting for working memory geeks: we show each other works-in-progress meant to generate new ideas for research, we go for a lovely hill-walk, and we cap off each day with a mid-summer evening stroll to the local pub. Though established and sustained by the core group of researchers advocating the famous multi-component model of working memory, others (even skeptics like me) are welcomed. I saw results that challenged my point of view, and maybe others with different points of view feel the same way. However, I left with the uncomfortable impression that one class of post-hoc explanation, the use of unanticipated mnemonic strategies, will always pop up and save the standard working memory story from useful criticism, criticism that might ultimately help make the explanation better.

After nearly every presentation at the WMDM2016, discussion centered on mnemonic strategies. How did participants approach the task? Was it possible that participants encoded visual images verbally and then serially rehearsed them? Perhaps they remembered verbal items by weaving a story out of them, or imagining them as pictures? In some sense, this is good methodology. Researchers should be mindful of the fact that human participants might find multiple ways to accomplish any task. This sort of analysis should happen, hopefully before much effort is spent in data collection, so that researchers have a sound understanding early on of the responses their research design affords. This is why so many researchers actually participate in their own studies during the piloting phases, and why we ask students of psychology to participate as subjects in an extensive variety of research studies before they design their own. However, post-hoc musing about flexible strategy use presents a big problem for advancing theory. It can prevent us from considering whether a theoretical idea needs modification. This has happened, and we need to be careful not to make a habit of it.

One vivid example comes from Shah and Miyake (1996). Shah and Miyake tested participants’ working memory spans in complex span tasks. Complex span tasks measure working memory span while forcing the participant to simultaneously perform an extra task: you are given some stimulus which you must make a decision about (such as whether a sentence makes sense or not, or whether a rotated letter is mirrored or not), then given a stimulus to remember, then given another stimulus to judge, then another to remember, and so on. The decision task keeps you occupied so that you cannot think much about the memory items until it is time to recall them. In their version, Shah and Miyake elegantly manipulated whether the memory items were verbal or spatial and also whether the extra decision task involved a verbal or spatial judgment. These manipulations were designed to test whether we needed to believe that verbal and spatial information is processed by distinct systems. If there are distinct systems, then verbal memory span should not be affected much by making spatial reasoning decisions, and likewise spatial memory span should not be affected much by making decisions about verbal stimuli.

Even with such a clear experimental design, the results were not quite in line with predictions. Verbal memory spans were clearly affected more by processing verbal stimuli than making spatial judgments. However, the evidence from spatial memory spans was puzzling: for some sequence lengths, it looked like spatial memory spans were lower when interrupted by the spatial task, but usually spatial memory spans looked equally bad regardless of which secondary task was used. Shah and Miyake (1996) explained this by invoking strategy use: possibly, participants tried to encode the spatial locations in words, making these representations susceptible to interference from the verbal decision task.

This is a perfectly plausible explanation for the observed results. But if true, it means that Shah and Miyake (1996) didn’t actually measure spatial memory very well, and perhaps they should not have drawn conclusions about it as though they had. If their strategic explanation were truly correct, rather than learning that verbal memories only get worse when paired with verbal tasks and spatial memories only get worse when paired with spatial tasks, they learned that verbal memories get worse when paired with verbal tasks and that spatial memories aren’t really spatial at all, or at least, they weren’t in this study for some of their participants. Appealing to strategy use should have led to the collection of new data in which the problem was rectified and spatial memory measured more convincingly. Instead, strategy use was invoked to avoid considering whether this complex pattern of results might actually mean that a different way of explaining working memory is needed.

The tenor of conversations about mnemonic strategies at WMDM2016 similarly seemed like attempts to avoid hard questions about cherished beliefs, not just constructive comments meant to improve measures of working memory. At some point, we need broad acknowledgement that post-hoc rationalizations about strategy use are not always merited and shouldn’t be uncritically given a lot of evidentiary weight.