Kushin Mukherjee¹ (), Timothy T. Rogers, Karen B. Schloss; ¹University of Wisconsin Madison, ²University of Wisconsin Madison, ³University of Wisconsin Madison

Color-concept associations are important for many facets of visual cognition from object recognition to interpretation of information visualizations. Thus, a key goal in vision science is developing efficient methods for estimating color-concept association distributions over color space. Such methods may also inform how people form associations between colors and abstract concepts despite these color-concept pairs never co-occurring in the natural world. To this end, we investigated the extent to which GPT-4, a multimodal large language model (LLM), could estimate human-like color-concept associations without any additional training. We first collected human association ratings between 70 concepts and a set of 71 colors spanning perceptual color space (UW-71 colors). We then queried GPT-4 to generate analogous ratings when given concepts as words and colors as hexadecimal codes, and compared these association ratings to the human data. Color-concept association ratings generated by GPT-4 were correlated with human ratings (mean r across concepts = .67) at a level comparable to state-of-the-art methods for automatically estimating such associations from images. The correlations between GPT-4 and human ratings varied across concepts (range: r = .08 – .93), with the correlation strength itself predicted by the specificity (inverse entropy) of the color-concept concept association distributions (r = .57, p < .001). Although GPT-4’s performance was also predicted by concept abstractness (r = -.42, p < .001), this effect was dominated by specificity when both factors were entered into a model together (specificity: p < .001, abstractness: p = .25). These results suggest that GPT-4 can be used as a tool for estimating associations between concepts and perceptual properties, like color, with better accuracy for high-specificity concepts. They also suggest that learning both word-to-percept structure and word-to-word structure, as multimodal LLMs do, might be one way to acquire associations between colors and abstract concept words.

Acknowledgements: NSF award BCS-1945303 to K.B.S.