TY - JOUR
T1 - Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions
AU - REPEAT Initiative
AU - Wang, Shirley V.
AU - Sreedhara, Sushama Kattinakere
AU - Schneeweiss, Sebastian
AU - Franklin, Jessica M.
AU - Gagne, Joshua J.
AU - Huybrechts, Krista F.
AU - Patorno, Elisabetta
AU - Jin, Yinzhu
AU - Lee, Moa
AU - Mahesri, Mufaddal
AU - Pawar, Ajinkya
AU - Barberio, Julie
AU - Bessette, Lily G.
AU - Chin, Kristyn
AU - Gautam, Nileesa
AU - Ortiz, Adrian Santiago
AU - Sears, Ellen
AU - Stefanini, Kristina
AU - Zakarian, Mimi
AU - Dejene, Sara
AU - Rogers, James R.
AU - Brill, Gregory
AU - Landon, Joan
AU - Lii, Joyce
AU - Tsacogianis, Theodore
AU - Vine, Seanna
AU - Garry, Elizabeth M.
AU - Gibbs, Liza R.
AU - Gierada, Monica
AU - Isaman, Danielle L.
AU - Payne, Emma
AU - Alwardt, Sarah
AU - Arlett, Peter
AU - Bartels, Dorothee B.
AU - Bate, Andrew
AU - Berlin, Jesse
AU - Bourke, Alison
AU - Bradbury, Brian
AU - Brown, Jeffrey
AU - Burnett, Karen
AU - Brennan, Troyen
AU - Chan, K. Arnold
AU - Choi, Nam Kyong
AU - de Vries, Frank
AU - Eichler, Hans Georg
AU - Filion, Kristian B.
AU - Freeman, Lisa
AU - Hallas, Jesper
AU - Happe, Laura
AU - Hennessy, Sean
N1 - Publisher Copyright:
© 2022, The Author(s).
PY - 2022/12
Y1 - 2022/12
N2 - Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.
AB - Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.
UR - http://www.scopus.com/inward/record.url?scp=85137010909&partnerID=8YFLogxK
U2 - 10.1038/s41467-022-32310-3
DO - 10.1038/s41467-022-32310-3
M3 - Article
C2 - 36045130
AN - SCOPUS:85137010909
SN - 2041-1723
VL - 13
JO - Nature Communications
JF - Nature Communications
IS - 1
M1 - 5126
ER -