From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

Mehar Bhatia, Sahithya Ravi*, Aditya Chinchure*, Eunjeong Hwang, Vered Shwartz
University of British Columbia, Vector Institute of AI
Published at EMNLP'24 Main
*Indicates Equal Contribution
Paper Code Data (Task1 Retrieval on HF) Data (Task2 Grounding on HF)

Abstract

Despite recent advancements in vision-language models, their performance remains suboptimal on images from non-western cultures due to underrepresentation in training datasets. Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures and do not adequately assess cultural diversity across universal as well as culture-specific local concepts. To address these limitations, we introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding. The former task entails retrieving culturally diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries. Our evaluation across a wide range of models reveals that the performance varies significantly across cultures -- underscoring the necessity for enhancing multicultural understanding in vision-language models.

An example instance from each task in GlobalRG benchmark: (i) Retrieval Across Universals measures the ability of VLMs to retrieve culturally diverse images for a query q. (ii) Cultural Visual Grounding aims to evaluate the ability of VLMs to identify a cultural concept q.

Research Questions Discussed

  1. Are able to (TASK 1) retrieve relevant and culturally diverse images for universals AND (TASK2) ground culture-specific local concepts?
  2. Do VLMs exhibit biases towards images from specific cultures?
  3. What are the challenges faced by VLMs in achieving (Task 1) high cultural diversity AND (TASK 2) grounding culture-specific concepts??

Task 1: Retrieval Across Universals


Task Definition

Let Q={q1,q2,…,qn} be a set of textual queries representing universal concepts, and I={I1,I2,…,Im} the set of images from different cultures. Given a query q∈Q, the goal is to retrieve a ranked list of images R(q,I)={Ir1,Ir2,…,Irk}⊂I that maximizes both relevance and cultural diversity.

Relevance: Rel(q,I) refers to how well the image I matches the query q captured by the standard precision@k.
Diversity: Div(R(q,I)) measures the cultural diversity of the retrieved images using the formula,

diversity@k=−1log⁡m∑i=1mpilog⁡(pi)

where pi is the proportion of images from the i-th culture in the top k retrieved images R(q),
and m is the total number of cultures in the top k.

A high entropy value (∼ 100) indicates high diversity, retrieved images are well-distributed across different cultures.
Conversely, a low entropy value (∼ 0) indicates low diversity, retrieved images are biased towards specific cultures.


Dataset Statistics

First Image

List of 50 cultures covered in the retrieval task.

Second Image

Human universals used as textual queries in 'Retrieval across Universals' task.



Model Performance

First Image

Average performance of various VLMs on the Retrieval across universals task, in terms of Relevance and Diversity.



Top-5 images retrieved for a sample of universals by models CLIP, CoCA and BLIP-2.

Task 2: Cultural Visual Grounding



Dataset Statistics

First Image

Detailed statistics of annotated images across different cultural groups and regions for Cultural Visual Grounding task.



Model Performance

First Image

Country-level Accuracy of each model on Cultural Visual Grounding task.

Second Image

Country group-level Accuracy of eachmodel on Cultural Visual Grounding task.



Examples

First Image

Citation

If you found this work useful in your own research, please consider citing the following:


        @article{bhatia2024local,
          title={From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models},
          author={Bhatia, Mehar and Ravi, Sahithya and Chinchure, Aditya and Hwang, Eunjeong and Shwartz, Vered},
          journal={arXiv preprint arXiv:2407.00263},
          year={2024}
        }