Unresolved heterogeneity in meta-analysis: Combined construct invalidity, confounding, and other challenges to understanding mean effect sizes

Timothy R. Levine, René Weber

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

We examined the interplay between how communication researchers use meta-analyses to make claims and the prevalence, causes, and implications of unresolved heterogeneous findings. Heterogeneous findings can result from substantive moderators, methodological artifacts, and combined construct invalidity. An informal content analysis of meta-analyses published in four elite communication journals revealed that unresolved between-study effect heterogeneity was ubiquitous. Communication researchers mainly focus on computing mean effect sizes, to the exclusion of how effect sizes in primary studies are distributed and of what might be driving effect size distributions. We offer four recommendations for future meta-analyses. Researchers are advised to be more diligent and sophisticated in testing for heterogeneity. We encourage greater description of how effects are distributed, coupled with greater reliance on graphical displays. We council greater recognition of combined construct invalidity and advocate for content expertise. Finally, we endorse greater awareness and improved tests for publication bias and questionable research practices.

Original languageEnglish
Pages (from-to)343-354
Number of pages12
JournalHuman Communication Research
Volume46
Issue number2-3
DOIs
StatePublished - 1 Apr 2020

Keywords

  • Construct Validity
  • Effect Heterogeneity
  • Effect Sizes
  • Meta-Analysis

Fingerprint

Dive into the research topics of 'Unresolved heterogeneity in meta-analysis: Combined construct invalidity, confounding, and other challenges to understanding mean effect sizes'. Together they form a unique fingerprint.

Cite this