Research by Princeton University neuroscientists provides a new framework for understanding how the experience of life is accumulated, stored and recalled by the human brain.
"When you go about your day, you're continuously assaulted with visual, auditory and other sensory information," said Christopher Baldassano, an associate research scholar at the Princeton Neuroscience Institute (PNI). "We don't try to understand our world as the continuous stream that's coming in, but we break it up into pieces we can understand and remember. The goal of this research was to look for the signatures of this kind of activity."
Baldassano and a team of researchers from the labs of Kenneth Norman and Uri Hasson at PNI and the Department of Psychology developed mathematical tools that, when applied to functional magnetic resonance imaging data, allowed them to detect this activity — known as chunking — happening as study participants watched a television show.
"The main result is that we do see this kind of chunking of events throughout the entire brain," Baldassano said. "So, for example, when you're watching a television show, many parts of your brain are involved in understanding what's going on. And in pretty much all these areas, we're seeing some kind of chunking. "
The researchers found that the chunks vary substantially in length, in a hierarchy that runs up from a few milliseconds in parts of the brain tied closely to sensory input to hundreds of seconds in parts of the brain where higher-order thinking takes place.
The researchers propose that sensory regions of the brain break experience into short chunks, as shown at the bottom of the illustration, which are then combined into longer and longer events by other brain regions, as seen at the top. These longer events can combine both audio and video information. When an event ends, it is stored in long-term memory.
The boundaries that mark the beginning and end of those longer chunks are coupled with increases in activity in the hippocampus, a region of the brain linked to memory, the researchers said. This suggests that "brain activity is naturally structured into nested events, which form the basis of long-term memory representations," the researchers wrote.
The research is detailed in an article titled "Discovering event structure in continuous narrative perception and memory" published Aug. 2 in the journal Neuron by Baldassano; Janice Chen, a former postdoctoral researcher at Princeton and now an assistant professor of psychology and brain sciences at Johns Hopkins University; Asieh Zadbood, a graduate student in psychology; Jonathan Pillow, an associate professor of psychology and PNI; Hasson, a professor of psychology and PNI; and Norman, a professor of psychology and PNI.
The researchers analyzed fMRI data for study participants who had watched an episode of the BBC television show "Sherlock" while having their brain activity recorded. The mathematical tools developed by the researchers identified sudden changes in brain activity in the different regions being monitored, signifying when one memory chunk ended and another began.
In the video below, brain activity of study participants was recorded using an fMRI, as shown at left. The recording is shown at 35 times actual speed. The highlighted brain region is the angular gyrus, a region found to construct high-level chunks lasting approximately one minute. At right, the moving dot represents the pattern of activity in angular gyrus, relative to the nine landmark patterns shown. The speed of the dot shows how quickly the brain activity pattern is changing. A quick jump indicates that the activity pattern changed rapidly. Dots with the same color belong to the same chunk, as determined by the model developed by the authors.
"Using this analysis method Chris developed, we can look everywhere in the brain simultaneously and try to figure out what each brain area thinks the chunks are and how they relate to each other," Norman said. "It asks the brain instead of having to ask people where they think one event ends and another begins."
Jeffrey Zacks, a professor of radiology at Washington University in St. Louis who studies perception and cognition, said the research represents an advance in efforts to understand how "the brain creates the conscious present."
"We all have the sense that there is a present moment that is more than an instant, but rather has some width and structure to it," said Zacks, who is familiar with the research but did not participate in it. "Baldassano and his colleagues have found evidence for neural representations that have these same properties — and discovered that they are organized in a sensible temporal hierarchy from shorter to longer time-scales.”
Baldassano said he hopes to continue studying the phenomenon of chunking to understand more about the brain patterns detected in fMRI data, such as whether the brain is constructing the chunks from scratch or applying templates from similar events in the past. In addition, he hopes to apply the research techniques to more and more realistic scenarios.
"How do you even begin to tackle the problem of how memory is working in a precise way?" Norman said. "The chunking algorithm Chris developed gives us moments we can dig into. We're seeing naturally how brains chunk information. And, drawing on what else we know about memory, we can see what brain structures we know are important to memory are doing at the moments this algorithm has identified. This gives us the ability to be very precise about how we analyze and think about these very rich, meaningful naturalistic sorts of data sets."
The mathematical tools developed by the Princeton researchers will be shared with other researchers in the Brain Imaging Analysis Kit (BrainIAK), which is part of a collaboration between Princeton researchers and colleagues at Intel Labs, a research arm of Intel Corp.
This work was supported by a grant from Intel Labs, the National Institutes of Health (R01 MH112357-01, R01-MH094480, 2T32MH065214-11), the McKnight Foundation, a National Science Foundation Career Award (IIS-1150186) and a grant from the Simons Collaboration on the Global Brain (SCGB AWD1004351).
