Reproducibility, a hallmark of science, is typically assessed in validation studies. We focus on high-throughput studies where a large number of biomarkers is measured in a training study, but only a subset of the most significant findings is selected and re-tested in a validation study. Our aim is to get the statistical measures of overall assessment for the selected markers, by integrating the information in both the training and validation studies. Naive statistical measures, such as the combined (Formula presented.) -value by conventional meta-analysis, that ignore the non-random selection are clearly biased, producing over-optimistic significance. We use the false-discovery rate (FDR) concept to develop a selection-adjusted FDR (sFDR) as an overall assessment measure. We describe the link between the overall assessment and other concepts such as replicability and meta-analysis. Some simulation studies and two real metabolomic datasets are considered to illustrate the application of sFDR in high-throughput data analyses.
Bibliographical noteFunding Information:
Donghwan Lee was supported by National Research Foundation of Korea Grants (NRF‐2021R1A2C1012865). Woojoo Lee was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea government (MSIT) (no. 2021R1A2C1014409).
© 2022 John Wiley & Sons Ltd.
- false discovery rate
- selection adjustment
- validation study