SemEval-2022 Task 8: Multilingual news article similarity

Xi Chen, Ali Zeynali, Chico Q. Camargo, Fabian Flock, Devin Gaffney, Przemyslaw A. Grabowicz, Scott A. Hale, David Jurgens, Mattia Samory

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

23 Scopus citations


Thousands of new news articles appear daily in outlets in different languages. Understanding which articles refer to the same story can not only improve applications like news aggregation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news articles is challenging due to the different dimensions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 correlation with gold standard labels across languages, human annotators are capable of reaching higher correlations, suggesting space for further progress.

Original languageEnglish
Title of host publicationSemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop
EditorsGuy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
PublisherAssociation for Computational Linguistics (ACL)
Number of pages13
ISBN (Electronic)9781955917803
StatePublished - 2022
Event16th International Workshop on Semantic Evaluation, SemEval 2022 - Seattle, United States
Duration: 14 Jul 202215 Jul 2022

Publication series

NameSemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop


Conference16th International Workshop on Semantic Evaluation, SemEval 2022
Country/TerritoryUnited States

Bibliographical note

Funding Information:
This research has received funding through the Volkswagen Foundation. We thank Media Cloud for data access. We thank the Internet Archive which made it possible for participants to all have access to the same data. We are deeply grateful to the annotators and task participants: thank you.

Publisher Copyright:
© 2022 Association for Computational Linguistics.


Dive into the research topics of 'SemEval-2022 Task 8: Multilingual news article similarity'. Together they form a unique fingerprint.

Cite this