Multisensory Research Group

Focus: The anatomical, physiological, behavioral, and computational bases for the development and expression of multisensory integration.

Decoding and interpreting incoming sensory information are among the brain's most important tasks. These are ongoing processes that make it possible for us not only to know the world in which we live, but to plan and initiate behaviors that are appropriate for a particular circumstance. Because survival depends on the speed and accuracy of such processes, it is not surprising to find that encoding, decoding, and evaluating sensory information have been powerful driving forces in evolution. Consequently, extant organisms have an impressive array of specialized sensory systems.

Having multiple sensory systems provides significant benefits; it allows an organism to monitor simultaneously a host of environmental cues, and also provides a means of substituting one sensory system for another when necessary (e.g., hearing and/or touch can substitute for vision in the dark). The ability to monitor and process multiple sensory cues in “parallel” not only increases the likelihood that a given stimulus will be detected, but, because the information carried along each sensory channel reflects a different feature of that stimulus, it increases the chances that the stimulus will be properly identified. Stimuli that may be difficult to distinguish by means of a single sensory modality (e.g., how they look) can become quite distinct via information from another modality (how they sound or feel).

Of particular benefit is the brain’s ability to use the information carried by these different sensory channels synergistically. Thus, different sensory inputs can enhance one another’s effectiveness, increasing the likelihood that an event will be detected, properly identified, and that an appropriate response is initiated as fast as is possible.

Our research goal is to determine how information from different senses is pooled in making decisions, and how the brain develops this remarkable capacity. We use two interrelated neural models to do this: one in the midbrain superior colliculus (SC) and one in association cortex. The midbrain SC has proved to be a most effective model, but its principal features have been shown also to be operative in cortex. Its advantage is not only its host of sensory inputs (visual, auditory, somatosensory), but its many multisensory neurons, and its distinct behavioral function: faciliatating the detection, localization and orientation to external events.

Current Projects:

Contrasts In How The Brain Integrates Within-Modal and Cross-Modal Information:Despite the substantial literature describing the impact of multisensory integration in multiple brain areas, and the importance of this process for perception and behavior, a fundamental question has been largely ignored: does the fusion of information from different sensory source yield a product that is different from the fusion of information from the same sensory source?  We are investigating this question at physiological and behavioral levels.

Anatomical Bases of Multisensory Integration: Physiological and behavioral studies have revealed that SC multisensory integration depends on the functional integrity of converging projections that descend from different regions of association cortex. We are currently investigating the nature of these projections and the development of this cortico-SC circuit using modern neuroanatomical tracing techniques and electron microscopy.

Computational Bases of Multisensory Integration: Recent evidence suggests that multisensory integration at the physiological level involves both linear and nonlinear computations.  We are currently developing and testing a neural network model that seeks to explain a multitude of empirical findings within a single framework.

Effects of Multisensory Integration on Response Timing and Information: The magnitude of the physiological impact of integration at the single neuron level has traditionally been measured as a change (usually expressed as a % change) in the total number of stimulus-elicited impulses.  We have recently found that concordant cross-modal stimuli not only produce more robust responses, but significantly speeds these responses.  In addition, we have found that integration produces substantial enhancements in information transmission; conveying not only more information, but information at a faster rate.

Similarities and Differences in Integration Across Species: The traditional animal model of SC multisensory integraiton is the cat, and the anatomical circuit underlying integration has been well-described in this model.  We have recently begun evaluating the generality of this anatomical circuit across mammalian species.  

Experiential and Anatomical Dependencies of the Development of Integration:Our recent evidence suggests that capacity to engage in multisensory integration is not innate capabiility, but rather one that the brain develops after birth in an experience-dependent fashion.  We are currently investigating the constraints guiding its appearance and maturation; in particular, the anatomical circuits involved, the essential properties of the cues that are critical to link them to one another, and whether the strategies of integration change during maturation.

Multisensory Integration in Motion Perception: Natural environments are dynamic, and salient targets are frequently in motion.  Animals interact appropriately with these targets by estimating their velocity and direction of movement.  We are current investigating how the brain combines information from different senses to affect and improve these decisions.

People:

Current Members

• Dr. Barry E. Stein - Professor and Chair. Interests: Multisensory integration, its circuitry and critical experiential and developmental antecedents.
• Dr. Ramanyran Ramachandran - Assistant Professor. Interests: Auditory signal processing, computational methods, motion perception.
• Dr. Benjamin Rowland - Assistant Professor. Interests: Computational bases of multisensory integration, temporal dynamics of integration.
• Dr. J. "Bill" Vaughan - Assistant Professor. Interests: Developmental antecedents of multisensory integration, physiological correlates of behavior.
• Dr. Thomas Perrault, Jr. – Assistant Professor. Cortical-subcortical interactions in multisensory integration.
• Dr. Juan Carlos Alvarado - Postdoctoral Fellow. Interests: How the brain integrates within- vs.cross-modal stimuli: physiology.
• Dr. Veronica Fuentes-Santamaria - Postdoctoral Fellow. Interests: Anatomical bases of multisensory integration, synaptic morphology.
• Dr. Guy Gingras - Postdoctoral Fellow. Interests: Behavioral assessment of within-modal and cross-modal integration.
• Dr. J. "Chad" Smith - Postdoctoral Fellow. Interests: Cross-species similarities and differences in multisensory integration.

Collaborators

• Dr. Terrence Stanford
• Dr. John McHaffie
• Dr. Paul Laurienti
• Dr. Mark Wallace (Vanderbilt)
• Dr. Thomas Anastasio (UIUC)
• Dr. Elisabetta Ladavas (University of Bologna) 

Educational Resources:

Books

• Stein, B.E. & Meredith, M.A. (1993).  The merging of the senses. MIT Press: Cambridge, MA.
• Calvert, G.A., Spence, C., & Stein, B.E. (2004). The handbook of multisensory processes. MIT Press: Cambridge, MA.

Recent Publications (2007): Updated 06/22/2007

Rowland BA, Stanford TR, and Stein BE (2007) A model of the neural mechanisms underlying multisensory integration in the superior colliculus.  Perception (In Press).

Jiang W, Jiang H, and Stein BE (2006) Neonatal cortical ablation disrupts multisensory development in the superior colliculus.  J. Neurophysiol. 95:1380-1396. [PDF]

• Department of Neurobiology and Anatomy