Prof. Kathy T. Mullen

Research Interests


* See my full publication list       * Link to my home page       * Home page of McGill Vision Research Labs

The overall aim of my research is to understand how the color content of the visual scene is encoded and analyzed within the human visual system. Initially, experimental approaches to color vision were primarily concerned with the very first stages of vision, namely with identifying the visual pigments that absorb light prior to the encoding of the visual scene. Although these first receptoral stages are a prerequisite for vision, they tell us little about how color is encoded, since the sensation of color is constructed post-receptorally by the responses of the color opponent neurons in the retina, visual pathways and the visual cortex of the brain.

Several decades ago a new approach emerged in the investigation of color vision based on the use of visual stimuli that uniquely activate the responses of the color sensitvie pathways in the visual system, so separating the brain's response to color from its response to black & white. Such stimuli (termed isoluminant) have become an important tool for the investigation of the neural basis of color vision. I have used these methods extensively. My general approach is to develop and test models and ideas of how the visual system encodes colour information using the behavioral testing of human vision (psychophysicics) and/or fMRI methods.  These can also be linked with available primate anatomical and physiological data.

Below are brief descriptions of some of the key issues that I have addressed in color vision research:

1. fMRI investigations of color processing in the human brain.
2. Color and motion processing - does the dorsal pathway contribute to color vision?
3. Color and form perception - defining the role of the ventral pathway in color vision.
4. Color vision across the visual field: is cone opponency lost in peripheral vision?
5. How do the cones combine into the opponent processes?
6. The cognitive development of color vision  in young children.

 


1. fMRI investigation of color vision in the human brain

Using retinotopic mapping and other localizing stimuli we define the visual areas of the human brain and its subthalamic nucleus the LGN.  We investigate the responses of these different visual areas to chromatic and black & white stimuli in order to define the role of each brain area in color processing and to define the way processing changes as information flows between the different areas.

Example publications for normal vision:

Mullen, K.T. Thompson, B. & Hess, R.F.  Responses of the human visual cortex and LGN to achromatic and chromatic temporal modulation: an fMRI study. Journal of Vision, 10(13): 13, 1–19, 2010. http://www.journalofvision.org/content/10/13/13.full.pdf


Mullen, K.T., Dumoulin, S.O. & Hess, R.F.  Color responses of the human lateral geniculate nucleus: Selective amplification of  S-cone signals between the lateral geniculate nucleus and primary visual cortex measured with high-field fMRI.  European Journal of Neuroscience, 28, 1911-1923, 2008. (PDF)

Mullen, K.T., Dumoulin, S.O., McMahon, K.L., de Zubicarary, G.I. & Hess, R.F.  Sensitivity of human retinotopic visual cortex to red-green, blue-yellow and achromatic contrast. European Journal of Neuroscience, 25, 491-502, 2007. (PDF)

Example publications in amblyopia:

Hess, R.F., Thompson, B., Gole, G. & Mullen, K.T.  The amblyopic deficit and its relationship to geniculo-cortical processing streams. Journal of Neurophysiology, 104(1), pp475-483, 2010. http://jn.physiology.org/cgi/reprint/104/1/475

Hess, R.F., Thompson, B., Gole, G. & Mullen, K.T.  Deficient responses from lateral geniculate nucleus in humans with amblyopia. European Journal of Neuroscience, 29, 1064-1070, 2009. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2695153/



 2. Motion processing in colour vision

Normal human color vision is known to be deficient in the perception of motion. This has lead to a 'text-book' view of the human cortex in which there are distinct 'color' and 'motion' streams: a color-blind dorsal stream that encodes motion information, and a motion-blind ventral stream that conveys color information. We are presently investigating using both psychophysical and fMRI methods the extent to which these streams are really separate for the processing of color and motion. This involves making a distinction between two types of motion: first order (linear) and higher order (nonlinear).  We find that color is defective in the processing of linear motion, but retains somecapacity to process non linear motion. One of the outstanding quesitons is where first and higher order motion are processed in the brain, and why only one of these types should be blind to color contrast.

Example publications:

Garcia-Suarez, L. & Mullen, K.T. Global motion processing in human color vision: a deficit for second order stimuli. Journal of Vision, 10(14): 20, 1–11, 2010. Free access: http://www.journalofvision.org/content/10/14/20.full.pdf

Michna, M. L. & Mullen, K.T.  The contribution of color to global motion. Journal of Vision, 8 (issue 5 article 10), 1-12, 2008. Free acess: JOV: http://journalofvision.org/8/5/10/Michna-2008-jov-8-5-10.pdf


Michna, M. L. & Mullen, K.T.  Chromatic mechanisms mediating motion for S-cone isolating stimuli. Vision Research, 47, 1042-1054, 2007. (PDF).

Mullen, K.T., Yoshizawa, T., & Baker, C.L. Luminance signals mediate the motion of red-green isoluminant gratings: the role of “temporal chromatic aberration”. Vision Research, 43, 1235-1247, 2003. (PDF)

Yoshizawa, T., Mullen, K.T. & Baker, C.L. Absence of a chromatic linear motion mechanism in human vision. Vision Research, 40, 1993-2010, 2000. (PDF)


 3. Color and form perception

For several decades color vision was considered very poor at seeing form and shape, and even called 'form blind'. This view was supported by phenomena such as the lowpass and low acuity nature of the color contrst sensitivity function (e.g. Mullen, 1985), the blurred appearance of chromatic borders, and by the loss of 3D shape perception under some conditions.  We (McIlhagga & Mullen, 1997) dubbed this view of color vision the "coloring book model" because it describes a subordinate role for color vision in the extraction of shape and form: color vision simply fills-in the contours and boundaries of objects that are primarily defined by luminance contrast (black & white). Subsequently, we have shown that this cannot be the main way the brain uses color.  Instead my results suggest that both color and luminance vision contribute to the primary stages of spatial processing in a very similar manner, with relatively little deficiency found for color vision. These results indicate suggest that color and luminance contrast may feed common form processing mechanisms at some stage in the visual pathway. So far, we find little evidence for a segregated pathway for color vision with poor form processing.


i. The processsing of orientation in color vision
: The encoding of orientation is a key part of spatial processing. Studies in my lab have revealed the role that color contrast plays in encoding orientation. We find color has only a mild deficiency of orientation processing. This is interesting as there is a populaton of chromatic neurons in the cortex that show very little orientation tuning. Our results therefore suggest that these are not part of a neural pathway encoding form in color vision but instead must have another role.

Beaudot, W.H.A. & Mullen, K.T.  Orientation selectivity in luminance and color vision assessed using 2-d bandpass filtered spatial noise. Vision Research, 45, 687-696, 2005.  (PDF)

Beaudot, W.H.A. & Mullen, K.T. Orientation discrimination in human vision: psychophysics and modeling. Vision Research, 46, 26-46, 2006. (PDF)

ii. The role of color in contour perception
: The detection of contours and object borders is fundamental to the extraction of the salient features of the visual scene.  We know that the perception of a border or contour depends on the integration or grouping of multiple outputs of many local neurons (detectors) in the visual system. I have used a contour integration task to compared the performance of the two color systems (red-green and blue-yellow) and the luminance (black & white) system in this role.   A black & white stimulus is illustrated below.


From Mullen, Beaudot & McIlhagga, 2000 (PDF). The wiggly path shown on the right is embedded in the figure shown on the left.

The left pannel contains a winding 'contour' made up of 10 aligned elements (Gabors) embedded in a background of randomly oriented Gabors.  The 'contour' alone is shown in the right pannel. Accurate detection of the contour relies on the integration of these multiple small oriented elements (Gabors) and requires the linking of the orientations of the contour elements across space. Our results reveal striking similarities in the way the color and luminance systems perform contour integration, in terms of their overall performances, efficiency, internal orientation noise, contrast dependence and dependence on curvature (McIlhagga & Mullen, 1996; Mullen, Beaudot & McIlhagga, 2000). We suggest that this indicates that color and luminance vision use a common neural process in the early cortical stages of form processing.

McIlhagga, W.H. & Mullen, K.T.  Contour integration with color and luminance contrast.  Vision Research, 36, 1265-1279, 1996. (PDF)

Mullen, K.T., Beaudot, W.H.A. & McIlhagga, W.H.   Contour integration in color vision: a common process for the blue-yellow, red-green and luminance mechanisms?   Vision Research, 40, 639-655, 2000. (PDF)

Beaudot, W.H.A. & Mullen, K.T.   Processing time of contour integration: the role of color, contrast and curvature.  Perception, 30, 833-853, 2001. (PDF)

Hess, R.F., Beaudot, W.H.A. & Mullen, K.T.  Dynamics of contour integration. Vision Research, 41, 1023-1037, 2001. (PDF)

Beaudot, W.H.A. & Mullen, K.T.  How long-range is contour integration in human color vision?  Visual Neuroscience, 20, 51-64, 2003. (PDF)


iii. Global shape processing: We have shown that color has a deficiency in a task used to test global shape perception (Mullen & Beaudot, 2002). Stimuli are concentric ring patterns termed radial frequency patterns, as shown on the right for red-green stimuli. The task is to discriminate of the shape of the pattern from circular (in this case a diamond shape from circular). For the top pair of chromatic stimuli, discrimination of the shape from the circle is relatively easy, but is harder for the lower pair.  The task is made progressively harder to determine the shape discrimination threshold. We find that red-green and particularly blue-yellow color vision shows a significant deficit on this task in comparison  to achromatic (black & white) vision. This is an interesting result as it may indicate that the specilialzed shape processing areas of the brain are better at using black & white than color contrast.  In further work we have addressed ideas on exactly how the shape discrimination is performed in order to understand what aspect is more poorly processes in color vision.

Mullen, K.T. & Beaudot, W.H.  The contribution of color vision to a global shape discrimination task. Vision Research, 42, 565-575, 2002. (PDF)

Mullen, K.T., Beaudot, W.H.A. & Ivanov, I.V.  Evidence that global processing does not limit thresholds for RF shape discrimination. Journal of Vision, 11(3): 3, pp1-21, 2011. http://www.journalofvision.org/content/11/3/6.full.pdf

iv. The early spatial filtering of colour vision: Earlier studies in my lab using sine-wave gratings and quantitative noise masking methods have shown that color vision, like luminance vision, filters the image piece-meal by arrays of orientationally selective bandpass spatial filters, and that these have a similar spatial bandwidth to those used by luminance vision.

Mullen, K.T. & Losada, M.A.  Evidence for separate pathways for color and luminance detection mechanisms.  Journal of the Optical Society of America A., 11, 3136-3151, 1994. (PDF)

Losada, M.A. & Mullen, K.T.  Color and luminance spatial tuning estimated by noise masking in the absence of off-frequency looking. Journal of the Optical Society of America A., 12, 250-260, 1995. (PDF)

Mullen, K.T. & Losada, M.A.  The spatial tuning of color and luminance peripheral vision measured with notch filtered noise masking. Vision Research, 39, 721-731, 1999. (PDF)  


 4. Color vision in the periphery and the "hit and miss" model of cone opponency

There are two color systems in primate vision: the 'blue-yellow', which uses S-cones and evolved several 100 million years ago, and the 'red-green' based L & M cones that selectively emerged in Old World primates about 30 million years. Until recently it was thought that both these cone opponent systems were carried by the P (parvo-cellular) cells of the retina and LGN, but physiological evidence from primates now suggests that the blue-yellow system has a private pathway from retina to cortex, using specialized retinal cells and the koniocellular layers of the LGN.  In human vision, I have established that the red-green cone opponent system is a foveal specialization: it is confined to central vision and red-green sensitivity declines much more steeply away from the fovea than does achromatic (luminance) sensitivity (Mullen 1991; Anderson, Hess & Mullen, 1991; Mullen & Kingdom, 1996). We have shown that the blue-yellow system has a very different distribution from red-green, showing a much more gradual decline across the visual field (Mullen & Kingdom, 2002).  This work thus supports the idea that the two cone opponent systems use separate neural pathways with different distributions across the visual field, and different rules of connectivity (Mullen & Kingdom, 2002).  It is important to note that these differences across the visual field originate post-receptorally and do not originate from any differences in the distributions of the cone receptors themselves.

There has been much debate about how red-green opponency is 'wired' into the visual system. Some groups proposed that red-green opponency arises as a by-product of the very small receptive fields found in the retinal P cells of central vision (see Mullen & Kingdom, 1996). The so-called 'hit and miss' hypothesis suggested that cone opponency can arise by chance in the small receptive fields of midget ganglion cells because, when cone numbers are low, proportions of L and M cones are likely to be distributed differentially to the centre and surround of a receptive field by chance. This implies that red-green cone opponency and fine visual acuity evolved in tandem in Old World primates, as midget retinal ganglion cells mediate both. Other groups suggested that, like blue-yellow opponency, RG opponency is selectively wired into large retinal neurons, and so is not dependent on small receptive fields and can extend into the peripheral visual field.  Mullen & Kingdom (1996) developed a quantitative model to define the loss in cone opponency expected as the cone numbers feeding a receptive field centre and surround increase. We find that the rapid loss in red-green cone opponency away from the fovea is consistent with the predictions of a random selection of cones feeding centre and surrounds of the midget ganglion cell receptive fields. This model of red-green cone opponency has also recently received direct physiological support in Macaque retina.

Sakurai, M. and Mullen, K.T. Cone weights for the two cone opponent systems in peripheral vision and asymmetries of cone contrast sensitivity. Vision Research, 46, 4346-4354, 2006. (PDF)

Mullen, K.T., Sakurai, M & Chu, W.  Does L/M cone opponency disappear in human periphery? Perception, 34, 951-959, 2005. (PDF)

Mullen, K.T. & Kingdom F.A.  Differential distribution of red-green and blue-yellow cone opponency across the visual field.  Visual Neuroscience, 19, 1-10, 2002. (PDF)

Mullen, K.T. & Kingdom F.A.  Losses in peripheral color sensitivity predicted from 'hit & miss' post-receptoral cone connections. Vision Research, 36, 1995-2000, 1996. (PDF)

Anderson S.A., Mullen, K.T. & Hess, R.F.  Human peripheral spatial resolution for achromatic and chromatic stimuli: Limits imposed by optical and retinal factors.  Journal of Physiology, 442, pp47-64, 1991. (http://www.ncbi.nlm.nih.gov/pubmed/1798037?dopt=Citation)

Mullen, K.T.  Colour vision as a post receptoral specialization of the central visual field.  Vision Research, 31, pp119-130, 1991. (PDF)
 


 5. Cone inputs to the cone opponent detection mechanisms

Behavioral (psychophysical) studies in human vision have revealed much about how the first neural stages of human color vision work.  We know that the red-green color system is based on the subtraction of the L from M cone responses (and vice versa), whereas the  blue-yellow system subtracts with S cone responses  from the combinatin of L and M cones, (and vice versa). In addition there is the luminance system, which adds the L and M cone responses. In a series of papers in the 1990s, I worked on providing quantitative results showing the relative weights of the cone contributions to these three systems.  I used two methods:  the measurement of detection threshold contours in a three dimensional cone contrast space fitted with super-ellipses (Sankeralli & Mullen, 1996) and the use of chromatic noise masking in a 3d cone contrast space (Sankeralli & Mullen, 1997). Our techniques have enabled us to improve on earlier estimates and we find that the red-green mechanism has a balanced L & M cone ratio with a small contribution from S cones, the luminance mechanism draws on L and M cones but is dominated by L cones and can have a small S cone contribution, and the blue-yellow mechanism is balanced with S cones in equal opposition to L and M cones. These results are important basis for establishing the so-called cardinal stimuli that allow the properties of the red-green, blue yellow and luminance post receptoral mechanisms to be investigated in isolation.

I have also established the independence of the two color and luminacne mechanisms at detection threshold using spatial summation (Mullen et al. 1997, Mullen & Sankeralli, 1999) and noise masking methods.

Sankeralli, M.J. & Mullen, K.T.  Estimation of the L-, M- and S-cone weights of the post-receptoral detection mechanisms. Journal of the Optical Society of America A, 13, 906-915, 1996. (PDF)

Sankeralli, M.J. & Mullen, K.T.  Postreceptoral chromatic detection mechanisms revealed by noise masking in three-dimensional cone contrast space. Journal of the Optical Society of America A. 14, 2633-2646, 1997. (PDF)

Mullen, K.T., Cropper, S.J. & Losada, M.A.  Absence of linear subthreshold summation between red-green and luminance mechanisms over a wide range of spatiotemporal conditions.  Vision Research, 37, 1167-1175, 1997. (PDF)

Mullen, K.T. & Sankeralli, M.J.  Evidence for the stochastic independence of the blue-yellow, red-green and luminance detection mechanisms revealed by subthreshold summation. Vision Research, 39, 733-745, 1999. (PDF)

Sankeralli, M.J. & Mullen, K.T.  A ratio model for suprathreshold hue increment detection. Journal of the Optical Society of America A., 16, 2625-2637, 1999. (PDF)

Sankeralli, M.J. & Mullen, K.T. Assumptions concerning orthogonality in threshold-scaled versus cone-contrast colour spaces. Vision Research, 41, 53-55, 2001. (PDF)

Sankeralli, M.J. & Mullen, K.T.   Bipolar or rectified chromatic detection mechanisms? Visual Neuroscience, 18, 127-135, 2001. (PDF)

Sankeralli, M.J., Mullen, K.T. & Hine, T.J.  Ratio model serves suprathreshold chromatic-luminance discrimination. Journal of the Optical Society of America A 18, 425-435, 2002. (PDF)



6.  The cognitive development of color vision in young children

I use color vision as a model for investigating how we establish internal or conceptual representations of perceptual properties. The sense of color involves not only the ability to perceive different colors but also the ability to recognize and identify colors. For example, consider being asked to find a red object in a room. This requires you to be able to see red but also to possess an internal concept of what 'red' is. If either one of these is lacking (percept or concept) the task is impossible. My interest in this area arose after I investigated a color agnosic, who could see colors perfectly but lacked the ability to identify them (Woodward et al. 1999). His color naming and color comprehension (pointing to a named color) were destroyed yet he could perceive color differences normally, had normal color contrast sensitivity, and could generate color words. This demonstrates a fundamental distinction between perceptual versus conceptual processing in the brain.

Young children pass through a stage in which they have difficulty establishing color concepts, as the following quote from Charles Darwin in 1877 about his own 4 children shows: "soon after they had come to the age when they knew the names of all common objects, I was startled by observing that they seemed quite incapable of affixing the right names to the colors in colored engravings, although I tried repeatedly to teach them." (from Bornstein, 1985).

The older literature (up to the 1980s) claims that reliable color naming develops surprisingly late at 4-7 years. Yet it is explicitly clear that this does not reflect a deficiency in color perception since infants can distinguish colors in the first few months of life, and color and luminance contrast sensitivity develop in parallel. Thus the delay lies in the development of color concepts, not perception, that underpin our ability to categorize and identify colors. In this sense, young children bear a striking resemblance to the adult color agnosic described above.  My research supports this view by showing that children have accurate color discrimination before they can reliably comprehend color terms. My research has explored two key questions:


1) Is color cognition selectively delayed in young children, or there in fact a more general delay in the conceptual development of all comparable visual attributes? Note that Darwin was comparing the learning of color words, which describe objects (ie. perceptual adjectives), to words for 'common objects' (ie. nouns). In a series of controlled psychophysical experiments on 47 children (2-6 years) we compared the development (comprehension and naming) of 4 visual attributes: form, size, speed, and color, all equated for visual saliency (Pitchford & Mullen, 2001). Results indicate no differential delays in the development of the comprehension and naming of these four visual attributes, suggesting that the delay for color so common reported in the literature may in fact may reflect a general delay in the visual cognition of object attributes. This new finding contradicts the historical contention that color is a special case in visual cognitive development and raises new and important questions for investigation.

2) What limits the development of color concepts? In a classic study Berlin & Kay (1969) made three predictions that radically altered the way we think about color processing. They postulated that there are 11 physiologically based basic perceptual color categories (red, green, blue, yellow, black, white, pink, purple, orange, brown and grey). They also claimed the evolutionary order by which color terms appear in world languages mirrors the salience of the conceptual representation of each color, and they predicted that children acquire color terms in the same order. While there is good support for the existence of 11 basic color categories, their other two predictions remain controversial. In an extensive study we addressed these important and unresolved issues in the development of color cognition (Pitchford & Mullen, 2002, Pitchford & Mullen, 2003).  We find that color names develop earlier than previously reported. Furthermore, we show that for most of the 11 colors there is no specific order for acquiring color names; nine of the 11 color names develop within a 3-month window with no predictable order. Hence we find no evidence that the 'primary' colors develop first, or in the order predicted by Berlin & Kay (1969). Curiously, we find a clear second stage of development with reliable naming and comprehension of brown and grey developing at least 6 months later than the other.  We do not believe this developmental order is driven by visual saliency as highly salient colors like red appear no earlier than less salient ones like blue or green.

Pitchford, N.J. & Mullen, K.T.  The role of perception, language and preference in the developmental acquisition of basic colour terms.  Journal of Experimental Child Psychology, 90, 275-302, 2005. (PDF)

Pitchford, N.J. & Mullen, K.T.  The development of conceptual colour categories in pre-school children: influence of perceptual categorization. Visual Cognition, 10, 51-77, 2003. (PDF)

Pitchford, N.J. & Mullen, K.T.  The developmental order of color term acquisition in young children. Perception, 31, 1349-1370, 2002. (PDF)

Pitchford, N.J. & Mullen, K.T.  Conceptualization of perceptual attributes: A special case for color?  Journal of Experimental Child Psychology, 80, 289-314, 2001. (PDF)


* Go back to my home page
KTM 15th March 2011