Maybe not so independent after all: The possibility, prevalence, and consequences of violating the independence assumptions in psychometric meta-analysis

Psychometric meta-analysis assumes that moderators are unrelated to study artifacts (e.g., criterion reliability), and that study artifacts are independent of true validities. Meeting these assumptions is important for researchers seeking to accurately partition the variance in effect sizes due to study artifacts from the variance due to meaningful moderators. Despite the critical role of these assumptions, we know very little about their tenability. To address this basic gap in the literature, we conducted three studies to determine if there are potential violations of the independence assumptions (Study 1), the prevalence of such violations (Study 2), and the consequences of violating the independence assumptions via a series of Monte Carlo simulations (Study 3). We found that violations of the independence assumptions are not only plausible but also routinely detected across a wide array of topics. Simulation results indicate that violating the independence assumptions can result in biases under certain circumstances, which are further accentuated due to the lack of stability in the estimators. We offer suggestions for the future use of psychometric meta-analysis and discuss the implications for research focused on refining psychometric meta-analysis.