Alexander Gokan^1,2, Arthur Shapiro^3,4; ¹Department of Neuroscience, American University, ²Center for Behavior and Neuroscience, American University, ³Department of Psychology, American University, ⁴Department of Computer Science, American University

How do color categories based strictly on visual information differ from those informed by language? We hypothesize two complementary ways that simple categorization schemes develop into complex categorizations schemes: “aligning” (existing categories converge to a shared location in perceptual space), and “refining” (amplifying and sharpening existing categorical patterns). We address this question with a representational similarity analysis (RSA) on 21 neural networks, some trained only on images and others on images and language. We measure representational alignment using a stimuli-normalized cosine similarity between activations extracted at every layer. Responses to pairs of stimuli (simple equiluminant patches on gray background) that differed from each other only in hue were compared across all angles on the CIELuv color circle to compute discriminability, from which we estimate the categorical prototypicality of hues. RSA analysis produces the following results 1.) The first layer of both language and non-language networks converge on categories centered on cyan, magenta, and yellow (attributable to optimal coding of input statistics), though residual networks show systematically different categories enabled by skip connections. 2.) Language provides immediate group consensus about color categories with minimal training, allowing models to take advantage of extended training to build more nuanced categories. 3.) Language stabilizes color representations across the models’ hierarchy, indicating more gradual transformation of information. 4.) language does not impose novel categorical structure, but instead amplifies minor categorical distinctions already present without language. Conclusion: RSA can disentangle two distinct processes (group alignment and categorical refinement) by which language facilitates color category development in neural networks. Our results show that language does not construct new categories, but rather stabilizes and amplifies existing, nonlinguistic categorical patterns. Lastly, we discuss how RSA can be extended to perform a variety of psychophysics experiments on neural networks, and demonstrate its applications to unique hues and optical illusions.