Eben Daggett¹ (), Giovanna Del Sordo¹, Jon Art¹, Michael Hout¹; ¹New Mexico State University

Recent studies have demonstrated the potential for improving Artificial Intelligence performance and alignment when utilizing human visual similarity judgements as training data. Computer vision models trained with human similarity data have exhibited visual representations of images that are better aligned with human visual representations than traditionally trained classifiers. Such models also show improved generalization abilities and, when errors are made, fail in a more predictable and less nonsensical manner. This finding has important implications in applications – e.g., the medical industry – where large, or nonsensical errors can be more deleterious than small errors. While such an approach to AI shows promise, it may be built on an unstable foundation, as the past 70 years of cognitive psychology research has demonstrated the construct of similarity to be highly flexible and difficult to quantify. In the present study, we trained CV models on human similarity judgments gathered for sets of medical pathology images and everyday household items, where the context of the similarity rating task was manipulated to induce context effects in the judgements. CV model performance in different conditions was analyzed to investigate how the flexible nature of visual similarity perception impacts an AI’s ability to effectively model human similarity data and generalize to exemplars not previously encountered by the AI. Results showed wide differentials in CV model performance between context conditions, but also demonstrated that model performance – in isolation – may not be an appropriate metric for assessing representational alignment. The study was intended as an initial examination into how similarity may play a better role in the improvement of AI visual classifiers for high-stakes applications. Recommendations are provided for future research on human-AI representational alignment so that the field may better operationalize its study via the inclusion of further research into the dynamics of human similarity perception.

Acknowledgements: Institute for Applied Practice in AI and Machine Learning - New Mexico State University