Research Interests

    The overall aim of my research is to understand how the color content of the visual scene is encoded and analyzed within the human visual system. Initially, experimental approaches to color vision were primarily concerned with the very first stages of vision, namely with identifying the visual pigments that absorb light prior to the encoding of the visual scene. Although these first receptoral stages are a prerequisite for vision, they tell us little about how color is encoded, since the sensation of color is constructed post�receptorally by combining the cone outputs into the responses of the color opponent neurons of the retina, visual pathways and the visual cortex of the brain. In order to separate and test these color responses in human vision, we use calibrated visual stimuli that can be chosen to selectively activate the individual cone responses, or the responses of the color sensitive and achromatic pathways in the visual system � in the letter case separating the brain's response to color from its response to black & white. Such stimuli have become an important tool for the investigation of the neural basis of color vision. My general approach is to develop and test models and ideas of how the visual system encodes colour information using the behavioral testing of human vision (psychophysics) often in combination with fMRI imaging methods. These can also be linked with available primate anatomical and physiological data. For a short review of my recent work see:

Johnson, E. N. & Mullen, K.T. Chapter: "Color in the Cortex" in the Springer Series in Vision Research, Vol. 5, Jan Kremers et al. (Eds): Human Color Vision. (Springer) 2016. Pp189-217. (PDF)
Mullen, K.T. The response to colour in the human visual cortex: the fMRI approach. Current Opinion in Behavioral Sciences, 30, pp 141-148, 2019. (PDF)

Below are brief descriptions of some of the key issues that I have addressed in color vision research:

1. fMRI investigations of color processing in the human brain.
2. Color and motion processing - does the dorsal pathway contribute to color vision?
3. Color and form perception - defining the role of the ventral pathway in color vision.
4. Color vision across the visual field: is cone opponency lost in peripheral vision?
5. How do the cones combine into the opponent processes?
6. The cognitive development of color vision in young children.

1. fMRI investigation of color vision in the human brain

    Using retinotopic mapping and other localizing stimuli we define the visual areas of the human brain and its subthalamic nucleus the LGN. In my lab, we investigate the responses of these different visual areas to chromatic and black & white stimuli in order to differentiate the roles of each brain area in color processing and to define the way processing changes as information flows between the different areas.

Example publications for normal vision:

Goddard, E. & Mullen, K.T. (2020) fMRI Representational Similarity Analysis reveals graded preferences for chromatic and achromatic stimulus contrast across human visual cortex, J. NeuroImage, 215, 116780, April 2020. DOI:10.1016/j.neuroimage.2020.116780 (PDF)

Goddard, E., Chang, D.H.F., Hess, R.F. & Mullen, K.T. Color contrast adaptation: fMRI fails to predict behavioral adaptation. NeuroImage, 201, pp1-13, 2019. (PDF)

Chang, D.H.F., Hess, R.F. & Mullen, K.T. Color responses and their adaptation in human superior colliculus and lateral geniculate nucleus. NeuroImage, 138, pp211-220 (2016). doi: 10.1016/j.neuroimage.2016.04.067. (PDF)

Mullen, K.T. Thompson, B. & Hess, R.F.  Responses of the human visual cortex and LGN to achromatic and chromatic temporal modulation: an fMRI study. Journal of Vision, 10(13): 13, 1–19, 2010. (PDF)

Mullen, K.T., Dumoulin, S.O. & Hess, R.F.  Color responses of the human lateral geniculate nucleus: Selective amplification of S-cone signals between the lateral geniculate nucleus and primary visual cortex measured with high-field fMRI. European Journal of Neuroscience, 28, 1911-1923, 2008. (PDF)

Mullen, K.T., Dumoulin, S.O., McMahon, K.L., de Zubicarary, G.I. & Hess, R.F.  Sensitivity of human retinotopic visual cortex to red-green, blue-yellow and achromatic contrast. European Journal of Neuroscience, 25, 491-502, 2007. (PDF)

Example publications in amblyopia:

Hess, R.F., Thompson, B., Gole, G. & Mullen, K.T.  The amblyopic deficit and its relationship to geniculo-cortical processing streams. Journal of Neurophysiology, 104(1), 475-483, 2010. (PDF)

Hess, R.F., Thompson, B., Gole, G. & Mullen, K.T.  Deficient responses from lateral geniculate nucleus in humans with amblyopia. European Journal of Neuroscience, 29, 1064-1070, 2009. (PDF)

2. Motion processing in colour vision

    Normal human color vision is known to be deficient in the perception of motion. This has led to a 'textbook' view of the human cortex in which there are distinct 'color' and 'motion' streams: a color-blind dorsal stream that encodes motion information, and a motion-blind ventral stream that conveys color information. In my lab, we have investigated using both psychophysical and rTMS methods the extent to which these streams are really separate for the processing of color and motion. rTMS allows for testing of the function of a human brain area by temporarily disrupting the neural signals with magnetic stimulation. In our psychophysical work, we make a distinction between two types of motion: first order (linear) and higher order (nonlinear). We find that color is defective in the processing of linear motion, but retains some capacity to process nonlinear motion. One of the outstanding questions is where first and higher order motion are processed in the brain, and why only one of these types should be blind to color contrast.

Example publications:

Cohen, D., Goddard, E. & Mullen, K.T. Re-evaluating hMT+ and hV4 functional specialization for motion and static contrast using fMRI-guided repetitive transcranial magnetic stimulation (rTMS). Journal of Vision 19(3):11, 1�20, 2019. doi:10.1167/19.3.11. (PDF)

Kaderali, S., Kim, Y.J., Reynaud, A. & Mullen, K.T. The role of human brain area hMT+ in the perception of global motion investigated with repetitive transcranial magnetic stimulation (rTMS). Brain Stimulation, 8(2), 200-207, 2015. (PDF)

Garcia-Suarez, L. & Mullen, K.T.  Global motion processing in human color vision: a deficit for second order stimuli. Journal of Vision, 10(14):20, 1–11, 2010. (PDF)

Michna, M. L. & Mullen, K.T.  The contribution of color to global motion. Journal of Vision, 8(5):10, 1-12, 2008. (PDF)

Michna, M. L., Yoshizawa, T. & Mullen, K.T.  S-cone contributions to linear and non-linear motion processing. Vision Research, 47, 1042-1054, 2007. (PDF)

Mullen, K.T., Yoshizawa, T., & Baker, C.L.  Luminance mechanisms mediate the motion of red-green isoluminant gratings: the role of "temporal chromatic aberration". Vision Research, 43, 1235-1247, 2003. (PDF)

Yoshizawa, T., Mullen, K.T. & Baker, C.L.  Absence of a chromatic linear motion mechanism in human vision. Vision Research, 40, 1993-2010, 2000. (PDF)

3. Color and form perception

    For several decades color vision was considered very poor at seeing form and shape, and was even called 'form blind'. This view was supported by phenomena such as the lowpass and low acuity nature of the color contrast sensitivity function (e.g. Mullen, 1985), the blurred appearance of chromatic borders, and by the loss of 3D shape perception under some conditions. We (McIlhagga & Mullen, 1997) dubbed this view of color vision the "coloring book model" because it describes a subordinate role for color vision in the extraction of shape and form: color vision simply fills-in the contours and boundaries of objects that are primarily defined by luminance contrast (black & white). Subsequently, we have shown that this cannot be the main way the brain uses color. Instead my results suggest that both color and luminance vision contribute to the primary stages of spatial processing in a very similar manner, with relatively little deficiency found for color vision. These results suggest that, for form perception, color and luminance contrast may feed common form processing mechanisms at some stage in the visual pathway. There may also be a different, segregated pathway for color surface perception with poor form processing.

i. The processing of orientation in color vision: The encoding of orientation is a key part of spatial processing. Studies in my lab have investigated the role that color contrast plays in encoding orientation. We find that overall human color vision has only a very mild deficiency in orientation discrimination. This is interesting as there is a population of highly colour sensitive neurons in area v1 of cortex that show very little orientation tuning. Our work also suggests that orientation tuning is less selective at low spatial frequencies, when this neuronal population is more active, but becomes more highly tuned and selective for orientation at mid-higher spatial frequencies, when the colour-luminance neurons of cortical area V1 are favoured. Our results therefore support the idea that there are two types of neural pathway in colour vision, one supporting form perception and another insensitive to form and edges, but potentially more sensitivity to the colour of surfaces.

Gheiratmand, M., Cherniawsky, A. & Mullen, K.T. The orientation tuning of binocular summation: a comparison of colour to achromatic contrast. Scientific Reports, 6, Article number: 25692, pp1-9, 2016. DOI: 10.1038/srep25692. (PDF)

Gheiratmand, M. & Mullen, K.T. Orientation tuning in human color vision at detection threshold. Scientific Reports, 4, Article number 4285, pp1-10, 2014. DOI: 10.1038/srep04285 (PDF)

Beaudot, W.H.A. & Mullen, K.T.  Orientation discrimination in human vision: psychophysics and modeling. Vision Research, 46, 26-46, 2006. (PDF)

Beaudot, W.H.A. & Mullen, K.T.  Orientation selectivity in luminance and color vision assessed using 2-d band-pass filtered spatial noise. Vision Research, 45, 687-696, 2005. (PDF)

ii. The role of color in contour perception: The detection of contours and object borders is fundamental to the extraction of the salient features of the visual scene. We know that the perception of a border or contour depends on the integration or grouping of multiple outputs of many local neurons (detectors) in the visual system. I have used a contour integration task to compare the performance of the two color systems (red-green and blue-yellow) and the luminance (black & white) system in this role. A black & white stimulus is illustrated below.

From Mullen, Beaudot & McIlhagga, 2000 (PDF). The wiggly path shown on the right is embedded in the figure shown on the left.

The left panel contains a winding 'contour' made up of 10 aligned elements (Gabors) embedded in a background of randomly oriented Gabors. The 'contour' alone is shown in the right panel. Accurate detection of the contour relies on the integration of these multiple small oriented elements (Gabors) and requires the linking of the orientations of the contour elements across space. Our results reveal striking similarities in the way the color and luminance systems perform contour integration, in terms of their overall performances, efficiency, internal orientation noise, contrast dependence, and dependence on curvature (McIlhagga & Mullen, 1996; Mullen, Beaudot & McIlhagga, 2000). We suggest that this indicates that color and luminance vision use a common neural process in the early cortical stages of form processing.

McIlhagga, W.H. & Mullen, K.T. Evidence for chromatic edge detectors in human vision using classification images. Journal of Vision, 18(9):8, 1-17, 2018. (PDF)

Kim, Y.J., Reynaud, A., Hess, R.F. and Mullen, K.T.  A normative data set for the clinical assessment of achromatic and chromatic contrast sensitivity using a qCSF approach. Invest Ophthalmol Vis Sci. 58(9): 3628-3636, 2017. DOI: 10.1167/iovs.17-21645. (PDF)

Beaudot, W.H.A. & Mullen, K.T.  How long range is contour integration in human color vision? Visual Neuroscience, 20, 51-64, 2003. (PDF)

Hess, R.F., Beaudot, W.H.A. & Mullen, K.T.  Dynamics of contour integration. Vision Research, 41, 1023-1037, 2001. (PDF)

Beaudot, W.H.A. & Mullen, K.T.  Processing time of contour integration: the role of colour, contrast and curvature. Perception, 30, 833-853, 2001. (PDF)

Mullen, K.T., Beaudot, W.H.A. & McIlhagga, W.H.  Contour integration in color vision: a common process for the blue-yellow, red-green and luminance mechanisms? Vision Research, 40, 639-655, 2000. (PDF)

McIlhagga, W.H. & Mullen, K.T.  Contour integration with colour and luminance contrast. Vision Research, 36, 1265-1279, 1996. (PDF)

iii. Global shape processing: We have shown that color has a deficiency in a task used to test global shape perception (Mullen & Beaudot, 2002). Stimuli are concentric ring patterns termed radial frequency patterns, as shown on the right for red-green stimuli. The task is to discriminate the shape of the pattern from circular (in this case a diamond shape from circular). For the top pair of chromatic stimuli, discrimination of the shape from the circle is relatively easy, but is harder for the lower pair. The task is made progressively harder to determine the shape discrimination threshold. We find that red-green and particularly blue-yellow color vision shows a significant deficit on this task in comparison to achromatic (black & white) vision. This is an interesting result as it may indicate that the specilialzed shape processing areas of the brain are better at using black & white than color contrast. In further work we have addressed ideas on exactly how the shape discrimination is performed in order to understand what aspect is more poorly processed in color vision.

Mullen, K.T., Beaudot, W.H.A. & Ivanov, I.V.  Evidence that global processing does not limit thresholds for RF shape discrimination. Journal of Vision, 11(3):3, 1-21, 2011. (PDF)

Mullen, K.T. & Beaudot, W.H.  Comparison of color and luminance vision on a global shape discrimination task. Vision Research, 42, 565-575, 2002. (PDF)

iv. The early spatial filtering of colour vision: Earlier studies in my lab using sine-wave gratings and quantitative noise masking methods have shown that color vision, like luminance vision, filters the image piece-meal by arrays of orientationally selective band-pass spatial filters, and that these have a similar spatial bandwidth to those used by luminance vision.

Mullen, K.T. & Losada, M.A.  The spatial tuning of color and luminance peripheral vision measured with notch filtered noise masking. Vision Research, 39, 721-731, 1999. (PDF)

Losada, M.A. & Mullen, K.T.  Color and luminance spatial tuning estimated by noise masking in the absence of off-frequency looking. Journal of the Optical Society of America A., 12, 250-260, 1995. (PDF)

Mullen, K.T. & Losada, M.A.  Evidence for separate pathways for color and luminance detection mechanisms. Journal of the Optical Society of America A., 11, 3136-3151, 1994. (PDF)

4. Color vision in the periphery and the "hit and miss" model of cone opponency

    There are two color systems in primate vision: the 'blue-yellow', which uses S-cones and evolved several 100 million years ago, and the 'red-green' based L & M cones that selectively emerged in Old World primates about 30 million years ago. Until recently it was thought that both these cone opponent systems were carried by the P (parvocellular) cells of the retina and LGN, but physiological evidence from primates now suggests that the blue-yellow system has a private pathway from retina to cortex, using specialized retinal cells and the koniocellular layers of the LGN. In human vision, I have established that the red-green cone opponent system is a foveal specialization: it is confined to central vision and red-green sensitivity declines much more steeply away from the fovea than does achromatic (luminance) sensitivity (Mullen 1991; Anderson, Hess & Mullen, 1991; Mullen & Kingdom, 1996). We have shown that the blue-yellow system has a very different distribution from red-green, showing a much more gradual decline across the visual field (Mullen & Kingdom, 2002). This work thus supports the idea that the two cone-opponent systems use separate neural pathways with different distributions across the visual field, and different rules of connectivity (Mullen & Kingdom, 2002). It is important to note that these differences across the visual field originate post-receptorally and do not originate from any differences in the distributions of the cone receptors themselves.

    There has been much debate about how red-green opponency is 'wired' into the visual system. Some groups proposed that red-green opponency arises as a by-product of the very small receptive fields found in the retinal P cells of central vision (see Mullen & Kingdom, 1996). The so-called 'hit and miss' hypothesis suggested that cone opponency can arise by chance in the small receptive fields of midget ganglion cells because, when cone numbers are low, proportions of L and M cones are likely to be distributed differentially to the centre and surround of a receptive field by chance. This implies that red-green cone opponency and fine visual acuity evolved in tandem in Old World primates, as midget retinal ganglion cells mediate both. Other groups suggested that, like blue-yellow opponency, RG opponency is selectively wired into large retinal neurons, and so is not dependent on small receptive fields and can extend into the peripheral visual field. Mullen & Kingdom (1996) developed a quantitative model to define the loss in cone opponency expected as the cone numbers feeding a receptive field centre and surround increase. We find that the rapid loss in red-green cone opponency away from the fovea is consistent with the predictions of a random selection of cones feeding centre and surrounds of the midget ganglion cell receptive fields. This model of red-green cone opponency has also recently received direct physiological support in Macaque retina.

Sakurai, M. and Mullen, K.T.  Cone weights for the two cone-opponent systems in peripheral vision and asymmetries of cone contrast sensitivity. Vision Research, 46, 4346-4354, 2006. (PDF)

Mullen, K.T., Sakurai, M & Chu, W.  Does L/M cone opponency disappear in human periphery? Perception, 34, 951-959, 2005. (PDF)

Mullen, K.T. & Kingdom F.A.  Differential distributions of red-green and blue-yellow cone opponency across the visual field. Visual Neuroscience, 19, 1-10, 2002. (PDF)

Mullen, K.T. & Kingdom F.A.  Losses in peripheral color sensitivity predicted from "hit & miss" post-receptoral cone connections. Vision Research, 36, 1995-2000, 1996. (PDF)

Anderson S.A., Mullen, K.T. & Hess, R.F.  Human peripheral spatial resolution for achromatic and chromatic stimuli: Limits imposed by optical and retinal factors.  Journal of Physiology, 442, 47-64, 1991. (PDF)

Mullen, K.T.  Colour vision as a post-receptoral specialization of the central visual field.  Vision Research, 31, 119-130, 1991. (PDF)

5. Cone inputs to the cone opponent detection mechanisms

    Behavioral (psychophysical) studies in human vision have revealed much about how the first neural stages of human color vision work. We know that the red-green color system is based on the subtraction of the L from M cone responses (and vice versa), whereas the blue-yellow system subtracts with S cone responses from the combinatin of L and M cones (and vice versa). In addition there is the luminance system, which adds the L and M cone responses. In a series of papers in the 1990s, I worked on providing quantitative results showing the relative weights of the cone contributions to these three systems. I used two methods: the measurement of detection threshold contours in a three dimensional cone contrast space fitted with super-ellipses (Sankeralli & Mullen, 1996) and the use of chromatic noise masking in a 3d cone contrast space (Sankeralli & Mullen, 1997). Our techniques have enabled us to improve on earlier estimates and we find that the red-green mechanism has a balanced L and M cone ratio with a small contribution from S cones, the luminance mechanism draws on L and M cones but is dominated by L cones and can have a small S cone contribution, and the blue-yellow mechanism is balanced with S cones in equal opposition to L and M cones. These results are important basis for establishing the so-called cardinal stimuli that allow the properties of the red-green, blue-yellow and luminance post-receptoral mechanisms to be investigated in isolation.

    I have also established the independence of the two color and luminance mechanisms at detection threshold using spatial summation (Mullen et al., 1997; Mullen & Sankeralli, 1999) and noise masking methods.

Sankeralli, M.J., Mullen, K.T. & Hine, T.J.  Ratio model serves suprathreshold color-luminance discrimination. Journal of the Optical Society of America A., 18, 425–435, 2002. (PDF)

Sankeralli, M.J. & Mullen, K.T. Assumptions concerning orthogonality in threshold-scaled versus cone-contrast colour spaces. Vision Research, 41, 53-55, 2001. (PDF)

Sankeralli, M.J. & Mullen, K.T.   Bipolar or rectified chromatic detection mechanisms? Visual Neuroscience, 18, 127-135, 2001. (PDF)

Mullen, K.T. & Sankeralli, M.J.  Evidence for the stochastic independence of the blue-yellow, red-green and luminance detection mechanisms revealed by subthreshold summation. Vision Research, 39, 733-745, 1999. (PDF)

Sankeralli, M.J. & Mullen, K.T.  Ratio model for suprathreshold hue-increment detection. Journal of the Optical Society of America A., 16, 2625-2637, 1999. (PDF)

Mullen, K.T., Cropper, S.J. & Losada, M.A.  Absence of linear subthreshold summation between red-green and luminance mechanisms over a wide range of spatio-temporal conditions.  Vision Research, 37, 1167-1175, 1997. (PDF)

Sankeralli, M.J. & Mullen, K.T.  Postreceptoral chromatic detection mechanisms revealed by noise masking in three-dimensional cone contrast space. Journal of the Optical Society of America A., 14, 2633-2646, 1997. (PDF)

Sankeralli, M.J. & Mullen, K.T.  Estimation of the L-, M- and S-cone weights of the postreceptoral detection mechanisms. Journal of the Optical Society of America A., 13, 906-915, 1996. (PDF)

6.  The cognitive development of color vision in young children

    I use color vision as a model for investigating how we establish internal or conceptual representations of perceptual properties. The sense of color involves not only the ability to perceive different colors but also the ability to recognize and identify colors. For example, consider being asked to find a red object in a room. This requires you to be able to see red but also to possess an internal concept of what 'red' is. If either one of these is lacking (percept or concept) the task is impossible. My interest in this area arose after I investigated a color agnosic, who could see colors perfectly but lacked the ability to identify them (Woodward et al. 1999). His color naming and color comprehension (pointing to a named color) were destroyed yet he could perceive color differences normally, had normal color contrast sensitivity, and could generate color words. This demonstrates a fundamental distinction between perceptual versus conceptual processing in the brain.

    Young children pass through a stage in which they have difficulty establishing color concepts, as the following quote from Charles Darwin in 1877 about his own 4 children shows: "Soon after they had come to the age when they knew the names of all common objects, I was startled by observing that they seemed quite incapable of affixing the right names to the colors in colored engravings, although I tried repeatedly to teach them." (from Bornstein, 1985).

    The older literature (up to the 1980s) claims that reliable color naming develops surprisingly late at 4-7 years. Yet it is explicitly clear that this does not reflect a deficiency in color perception since infants can distinguish colors in the first few months of life, and color and luminance contrast sensitivity develop in parallel. Thus the delay lies in the development of color concepts, not perception, that underpin our ability to categorize and identify colors. In this sense, young children bear a striking resemblance to the adult color agnosic described above. My research supports this view by showing that children have accurate color discrimination before they can reliably comprehend color terms. My research has explored two key questions:

1) Is color cognition selectively delayed in young children, or there is in fact a more general delay in the conceptual development of all comparable visual attributes? Note that Darwin was comparing the learning of color words, which describe objects (ie. perceptual adjectives), to words for 'common objects' (ie. nouns). In a series of controlled psychophysical experiments on 47 children (2-6 years) we compared the development (comprehension and naming) of 4 visual attributes: form, size, speed, and color, all equated for visual saliency (Pitchford & Mullen, 2001). Results indicate no differential delays in the development of the comprehension and naming of these four visual attributes, suggesting that the delay for color so commonly reported in the literature may reflect a general delay in the visual cognition of object attributes. This new finding contradicts the historical contention that color is a special case in visual cognitive development and raises new and important questions for investigation.

2) What limits the development of color concepts? In a classic study Berlin & Kay (1969) made three predictions that radically altered the way we think about color processing. They postulated that there are 11 physiologically based basic perceptual color categories (red, green, blue, yellow, black, white, pink, purple, orange, brown and grey). They also claimed the evolutionary order by which color terms appear in world languages mirrors the salience of the conceptual representation of each color, and they predicted that children acquire color terms in the same order. While there is good support for the existence of 11 basic color categories, their other two predictions remain controversial. In an extensive study we addressed these important and unresolved issues in the development of color cognition (Pitchford & Mullen, 2002, Pitchford & Mullen, 2003). We find that color names develop earlier than previously reported. Furthermore, we show that for most of the 11 colors there is no specific order for acquiring color names; nine of the 11 color names develop within a 3-month window with no predictable order. Hence we find no evidence that the 'primary' colors develop first, or in the order predicted by Berlin & Kay (1969). Curiously, we find a clear second stage of development with reliable naming and comprehension of brown and grey developing at least 6 months later than the other. We do not believe this developmental order is driven by visual saliency as highly salient colors like red appear no earlier than less salient ones like blue or green.

Pitchford, N.J. & Mullen, K.T.  The role of perception, language and preference in the developmental acquisition of basic color terms. Journal of Experimental Child Psychology, 90, 275-302, 2005. (PDF)

Pitchford, N.J. & Mullen, K.T.  The development of conceptual colour categories in pre-school children: influence of perceptual categorization. Visual Cognition, 10, 51-77, 2003. (PDF)

Pitchford, N.J. & Mullen, K.T.  Is the acquisition of basic-colour terms in young children constrained? Perception, 31, 1349-1370, 2002. (PDF)

Pitchford, N.J. & Mullen, K.T.  Conceptualization of perceptual attributes: a special case for color? Journal of Experimental Child Psychology, 80, 289-314, 2001. (PDF)