SemEval-2022 Task 8: Multilingual news article similarity

Xi Chen; Ali Zeynali; Chico Q. Camargo; Fabian Flock; Devin Gaffney; Przemyslaw A. Grabowicz; Scott A. Hale; David Jurgens; Mattia Samory

SemEval-2022 Task 8: Multilingual news article similarity

Xi Chen, Ali Zeynali, Chico Q. Camargo, Fabian Flock, Devin Gaffney, Przemyslaw A. Grabowicz, Scott A. Hale, David Jurgens, Mattia Samory

Division of English Language & Literature

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

40 Scopus citations

Abstract

Thousands of new news articles appear daily in outlets in different languages. Understanding which articles refer to the same story can not only improve applications like news aggregation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news articles is challenging due to the different dimensions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 correlation with gold standard labels across languages, human annotators are capable of reaching higher correlations, suggesting space for further progress.

Original language	English
Title of host publication	SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop
Editors	Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
Publisher	Association for Computational Linguistics (ACL)
Pages	1094-1106
Number of pages	13
ISBN (Electronic)	9781955917803
State	Published - 2022
Event	16th International Workshop on Semantic Evaluation, SemEval 2022 - Seattle, United States Duration: 14 Jul 2022 → 15 Jul 2022

Publication series

Name	SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop

Conference

Conference	16th International Workshop on Semantic Evaluation, SemEval 2022
Country/Territory	United States
City	Seattle
Period	14/07/22 → 15/07/22

Bibliographical note

Funding Information:
This research has received funding through the Volkswagen Foundation. We thank Media Cloud for data access. We thank the Internet Archive which made it possible for participants to all have access to the same data. We are deeply grateful to the annotators and task participants: thank you.

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Cite this

Chen, X., Zeynali, A., Camargo, C. Q., Flock, F., Gaffney, D., Grabowicz, P. A., Hale, S. A., Jurgens, D., & Samory, M. (2022). SemEval-2022 Task 8: Multilingual news article similarity. In G. Emerson, N. Schluter, G. Stanovsky, R. Kumar, A. Palmer, N. Schneider, S. Singh, & S. Ratan (Eds.), SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop (pp. 1094-1106). (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

Chen, Xi ; Zeynali, Ali ; Camargo, Chico Q. et al. / SemEval-2022 Task 8 : Multilingual news article similarity. SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. editor / Guy Emerson ; Natalie Schluter ; Gabriel Stanovsky ; Ritesh Kumar ; Alexis Palmer ; Nathan Schneider ; Siddharth Singh ; Shyam Ratan. Association for Computational Linguistics (ACL), 2022. pp. 1094-1106 (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

@inproceedings{1cddad6bffff40df8a4bff96ad6bd720,

title = "SemEval-2022 Task 8: Multilingual news article similarity",

abstract = "Thousands of new news articles appear daily in outlets in different languages. Understanding which articles refer to the same story can not only improve applications like news aggregation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news articles is challenging due to the different dimensions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 correlation with gold standard labels across languages, human annotators are capable of reaching higher correlations, suggesting space for further progress.",

author = "Xi Chen and Ali Zeynali and Camargo, {Chico Q.} and Fabian Flock and Devin Gaffney and Grabowicz, {Przemyslaw A.} and Hale, {Scott A.} and David Jurgens and Mattia Samory",

note = "Funding Information: This research has received funding through the Volkswagen Foundation. We thank Media Cloud for data access. We thank the Internet Archive which made it possible for participants to all have access to the same data. We are deeply grateful to the annotators and task participants: thank you. Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 16th International Workshop on Semantic Evaluation, SemEval 2022 ; Conference date: 14-07-2022 Through 15-07-2022",

year = "2022",

language = "English",

series = "SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1094--1106",

editor = "Guy Emerson and Natalie Schluter and Gabriel Stanovsky and Ritesh Kumar and Alexis Palmer and Nathan Schneider and Siddharth Singh and Shyam Ratan",

booktitle = "SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop",

}

Chen, X, Zeynali, A, Camargo, CQ, Flock, F, Gaffney, D, Grabowicz, PA, Hale, SA, Jurgens, D & Samory, M 2022, SemEval-2022 Task 8: Multilingual news article similarity. in G Emerson, N Schluter, G Stanovsky, R Kumar, A Palmer, N Schneider, S Singh & S Ratan (eds), SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 1094-1106, 16th International Workshop on Semantic Evaluation, SemEval 2022, Seattle, United States, 14/07/22.

SemEval-2022 Task 8: Multilingual news article similarity. / Chen, Xi; Zeynali, Ali; Camargo, Chico Q. et al.
SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. ed. / Guy Emerson; Natalie Schluter; Gabriel Stanovsky; Ritesh Kumar; Alexis Palmer; Nathan Schneider; Siddharth Singh; Shyam Ratan. Association for Computational Linguistics (ACL), 2022. p. 1094-1106 (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SemEval-2022 Task 8

T2 - 16th International Workshop on Semantic Evaluation, SemEval 2022

AU - Chen, Xi

AU - Zeynali, Ali

AU - Camargo, Chico Q.

AU - Flock, Fabian

AU - Gaffney, Devin

AU - Grabowicz, Przemyslaw A.

AU - Hale, Scott A.

AU - Jurgens, David

AU - Samory, Mattia

N1 - Funding Information: This research has received funding through the Volkswagen Foundation. We thank Media Cloud for data access. We thank the Internet Archive which made it possible for participants to all have access to the same data. We are deeply grateful to the annotators and task participants: thank you. Publisher Copyright: © 2022 Association for Computational Linguistics.

PY - 2022

Y1 - 2022

N2 - Thousands of new news articles appear daily in outlets in different languages. Understanding which articles refer to the same story can not only improve applications like news aggregation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news articles is challenging due to the different dimensions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 correlation with gold standard labels across languages, human annotators are capable of reaching higher correlations, suggesting space for further progress.

AB - Thousands of new news articles appear daily in outlets in different languages. Understanding which articles refer to the same story can not only improve applications like news aggregation but enable cross-linguistic analysis of media consumption and attention. However, assessing the similarity of stories in news articles is challenging due to the different dimensions in which a story might vary, e.g., two articles may have substantial textual overlap but describe similar events that happened years apart. To address this challenge, we introduce a new dataset of nearly 10,000 news article pairs spanning 18 language combinations annotated for seven dimensions of similarity as SemEval 2022 Task 8. Here, we present an overview of the task, the best performing submissions, and the frontiers and challenges for measuring multilingual news article similarity. While the participants of this SemEval task contributed very strong models, achieving up to 0.818 correlation with gold standard labels across languages, human annotators are capable of reaching higher correlations, suggesting space for further progress.

UR - http://www.scopus.com/inward/record.url?scp=85129313753&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85129313753

T3 - SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop

SP - 1094

EP - 1106

BT - SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop

A2 - Emerson, Guy

A2 - Schluter, Natalie

A2 - Stanovsky, Gabriel

A2 - Kumar, Ritesh

A2 - Palmer, Alexis

A2 - Schneider, Nathan

A2 - Singh, Siddharth

A2 - Ratan, Shyam

PB - Association for Computational Linguistics (ACL)

Y2 - 14 July 2022 through 15 July 2022

ER -

Chen X, Zeynali A, Camargo CQ, Flock F, Gaffney D, Grabowicz PA et al. SemEval-2022 Task 8: Multilingual news article similarity. In Emerson G, Schluter N, Stanovsky G, Kumar R, Palmer A, Schneider N, Singh S, Ratan S, editors, SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2022. p. 1094-1106. (SemEval 2022 - 16th International Workshop on Semantic Evaluation, Proceedings of the Workshop).

SemEval-2022 Task 8: Multilingual news article similarity

Abstract

Publication series

Conference

Bibliographical note

Fingerprint

Cite this