💡 About "Multilingual Benchmark MMLU Leaderboard"

Overview

The Multilingual Massive Multitask Language Understanding (MMMLU) benchmark is a comprehensive evaluation platform designed to assess the general knowledge capabilities of AI models across a wide range of domains. It includes a series of Question Answering (QA) tasks across 57 distinct domains, ranging from elementary-level knowledge to advanced professional subjects such as law, physics, history, and computer science.

Translation Effort

For this evaluation, we used the OpenAI MMMLU dataset, which has been extensively curated and tested for a multilingual understanding of AI models. The dataset includes 14 different languages and is specifically designed to assess how well AI models can handle a wide range of general knowledge tasks across 57 domains.

While the translation of the test set was performed by OpenAI, it ensures a high level of accuracy and reliability for evaluating multilingual models. By leveraging this pre-existing, professionally curated dataset, we aim to focus on model performance across multiple languages, without the need for additional translations from our side.

Commitment to Multilingual AI

By focusing on human-powered translations and publishing both the translated test sets and evaluation code, we aim to promote the development of AI models that can handle multilingual tasks with greater accuracy. This reflects our commitment to improving AI’s performance in underrepresented languages and making technology more inclusive and effective globally.

Locales Covered

The MMMLU benchmark includes a test set translated into the following locales:

• AR_XY: Arabic
• BN_BD: Bengali
• DE_DE: German
• ES_LA: Spanish
• FR_FR: French
• HI_IN: Hindi
• ID_ID: Indonesian
• IT_IT: Italian
• JA_JP: Japanese
• KO_KR: Korean
• PT_BR: Brazilian Portuguese
• SW_KE: Swahili
• YO_NG: Yoruba
• ZH_CN: Simplified Chinese

Purpose

The MMMLU Leaderboard aims to provide a unified benchmark for comparing AI model performance across these multiple languages and diverse domains. With the inclusion of the QA task across 57 domains, it evaluates how well models perform in answering general knowledge questions in multiple languages, ensuring a high standard of multilingual understanding and reasoning.

Goals

Our primary goal is to provide a reliable comparison for AI models across different languages and domains, helping developers and researchers evaluate and improve their models’ multilingual capabilities. By emphasizing high-quality translations and including a broad range of topics, we strive to make AI models more robust and useful across diverse communities worldwide.

🤗 How it works

Submit a model for automated evaluation on our clusters on the "Submit here" tab!

📈 Tasks

We evaluate models on a variety of key benchmarks, with a focus on Multilingual Massive Multitask Language Understanding (MMLU) and its variants, including MMLU, C-MMLU, ArabicMMLU, KoreanMMLU, MalayMMLU, and others. These benchmarks assess general knowledge across a wide range of topics from 57 categories, such as law, physics, history, and computer science.

The evaluation is performed using the OpenCompass framework, a unified platform for evaluating language models across multiple tasks. OpenCompass allows us to execute these evaluations efficiently and at scale, covering multiple languages and benchmarks.

For detailed information on the tasks, please refer to the "Tasks" tab in the OpenCompass framework.

Notes:

• The evaluations are all 5-shot.
• Results are aggregated by calculating the average of all the tasks for a given language.

🔎 Results

You can find:

• Detailed numerical results in the results dataset
• Community queries and running status in the requests dataset

✅ Reproducibility

To reproduce the results, you can use opencompass, Since many open-source models cannot fully adhere to instructions for QA tasks, we perform post-processing on the results by using Qwen2.5-7B-Instruct to extract the answers from the model's output. This is a relatively simple task, so we can generally extract the model's true output, which corresponds to options A, B, C, and D. As not all of our PRs are currently integrated into the main repository.

git clone git@github.com:BobTsang1995/opencompass.git
cd opencompass
pip install -e .
pip install lmdeploy
python run.py --models lmdeploy_qwen2_7b_instruct --datasets mmmlu_gen_5_shot -a lmdeploy

🙌 Acknowledgements

This leaderboard was independently developed as a non-profit initiative with the support of several academic institutions, which provided valuable assistance to make this effort possible. We extend our heartfelt gratitude to these institutions for their support.

• Technische Universität München (TUM)
• Tsinghua University
• Universiteit van Amsterdam
• Mohamed Bin Zayed University of Artificial Intelligence
• University of Macau
• Cardiff University
• Nara Institute of Science and Technology
• Shanghai Jiao Tong University
• Dublin City University
• Université Grenoble Alpes
• Universidade de Coimbra
• The Ohio State University
• RMIT University

The entities above are ordered chronologically by the date they joined the project. However, the logos in the footer are ordered by the number of datasets donated.

Thank you in particular to: Yi Zhou (Cardiff University), Yusuke Sakai (Nara Institute of Science and Technology), Yongxin Zhou (Université Grenoble Alpes), Haonan Li (MBZUAI), Jiahui Geng (MBZUAI), Qing Li (MBZUAI), Wenxi Li (Tsinghua University/Shanghai Jiaotong University), Yuanyu Lin (University of Macau), Andy Way (Dublin City University), Zhuang Li (RMIT University), Zhongwei Wan (The Ohio State University), Di Wu (University of Amsterdam), Wen Lai (Technical University of Munich) (TUM)

For information about the dataset authors please check the corresponding Dataset Cards (linked in the "Tasks" tab) and papers (included in the "Citation" section below). We would like to specially thank the teams that created or open-sourced their datasets specifically for the leaderboard (in chronological order):

• MMMLU: OpenAI

We also thank MacroPolo Team, Alibaba International Digital Commerce for sponsoring the inference GPUs.

🚀 Collaborate!

We would like to create a leaderboard as diverse as possible, reach out if you would like us to include your evaluation dataset!

Comments and suggestions are more than welcome! Visit the 👏 Multilingual-MMLU-Benchmark-Leaderboard discussions page, tell us what you think about MMMLU Leaderboard and how we can improve it, or go ahead and open a PR!

Thank you very much! 💛