Calculating Power for Multilevel Implementation Trials in Mental Health: Meaningful Effect Sizes, Intraclass Correlation Coefficients, and Proportions of Variance Explained by Covariates

Overview

Journal Implement Res Pract

Publisher Sage Publications

Date 2024 Sep 30

PMID 39346518

Authors

Nathaniel J Williams

Nicholas C Cardamone

Rinad S Beidas

Steven C Marcus

Affiliations

Soon will be listed here.

Abstract

Background: Despite the ubiquity of multilevel sampling, design, and analysis in mental health implementation trials, few resources are available that provide reference values of design parameters (e.g., effect size, intraclass correlation coefficient [ICC], and proportion of variance explained by covariates [covariate ]) needed to accurately determine sample size. The aim of this study was to provide empirical reference values for these parameters by aggregating data on implementation and clinical outcomes from multilevel implementation trials, including cluster randomized trials and individually randomized repeated measures trials, in mental health. The compendium of design parameters presented here represents plausible values that implementation scientists can use to guide sample size calculations for future trials.

Method: We searched NIH RePORTER for all federally funded, multilevel implementation trials addressing mental health populations and settings from 2010 to 2020. For all continuous and binary implementation and clinical outcomes included in eligible trials, we generated values of effect size, ICC, and covariate at each level via secondary analysis of trial data or via extraction of estimates from analyses in published research reports. Effect sizes were calculated as Cohen ; ICCs were generated via one-way random effects ANOVAs; covariate estimates were calculated using the reduction in variance approach.

Results: Seventeen trials were eligible, reporting on 53 implementation and clinical outcomes and 81 contrasts between implementation conditions. Tables of effect size, ICC, and covariate are provided to guide implementation researchers in power analyses for designing multilevel implementation trials in mental health settings, including two- and three-level cluster randomized designs and unit-randomized repeated-measures designs.

Conclusions: Researchers can use the empirical reference values reported in this study to develop meaningful sample size determinations for multilevel implementation trials in mental health. Discussion focuses on the application of the reference values reported in this study.

References

Brown A, Li P, Bohan Brown M, Kaiser K, Keith S, Oakes J . Best (but oft-forgotten) practices: designing, analyzing, and reporting cluster randomized controlled trials. Am J Clin Nutr. 2015; 102(2):241-8. PMC: 4515862. DOI: 10.3945/ajcn.114.105072. View

Elley C, Kerse N, Chondros P, Robinson E . Intraclass correlation coefficients from three cluster randomised controlled trials in primary and residential health care. Aust N Z J Public Health. 2005; 29(5):461-7. DOI: 10.1111/j.1467-842x.2005.tb00227.x. View

Davis-Plourde K, Taljaard M, Li F . Sample size considerations for stepped wedge designs with subclusters. Biometrics. 2021; 79(1):98-112. PMC: 9054939. DOI: 10.1111/biom.13596. View

Addington E, Cheung E, Bassett S, Kwok I, Schuette S, Shiu E . The MARIGOLD study: Feasibility and enhancement of an online intervention to improve emotion regulation in people with elevated depressive symptoms. J Affect Disord. 2019; 257:352-364. PMC: 6711819. DOI: 10.1016/j.jad.2019.07.049. View

Wilfley D, Agras W, Fitzsimmons-Craft E, Bohon C, Eichen D, Welch R . Training Models for Implementing Evidence-Based Psychological Treatment: A Cluster-Randomized Trial in College Counseling Centers. JAMA Psychiatry. 2019; 77(2):139-147. PMC: 6865264. DOI: 10.1001/jamapsychiatry.2019.3483. View

Korevaar E, Kasza J, Taljaard M, Hemming K, Haines T, Turner E . Intra-cluster correlations from the CLustered OUtcome Dataset bank to inform the design of longitudinal cluster trials. Clin Trials. 2021; 18(5):529-540. DOI: 10.1177/17407745211020852. View

Ivers N, Taljaard M, Dixon S, Bennett C, McRae A, Taleban J . Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000-8. BMJ. 2011; 343:d5886. PMC: 3180203. DOI: 10.1136/bmj.d5886. View

Waxmonsky J, Kilbourne A, Goodrich D, Nord K, Lai Z, Laird C . Enhanced fidelity to treatment for bipolar disorder: results from a randomized controlled implementation trial. Psychiatr Serv. 2013; 65(1):81-90. PMC: 4155734. DOI: 10.1176/appi.ps.201300039. View

Cook C, Lyon A, Locke J, Waltz T, Powell B . Adapting a Compilation of Implementation Strategies to Advance School-Based Implementation Research and Practice. Prev Sci. 2019; 20(6):914-935. PMC: 8943907. DOI: 10.1007/s11121-019-01017-1. View

10.

Ouyang Y, Li F, Preisser J, Taljaard M . Sample size calculators for planning stepped-wedge cluster randomized trials: a review and comparison. Int J Epidemiol. 2022; 51(6):2000-2013. PMC: 9749719. DOI: 10.1093/ije/dyac123. View

11.

Novins D, Green A, Legha R, Aarons G . Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. J Am Acad Child Adolesc Psychiatry. 2013; 52(10):1009-1025.e18. PMC: 3922894. DOI: 10.1016/j.jaac.2013.07.012. View

12.

Jackson C, Herschell A, Scudder A, Hart J, Schaffner K, Kolko D . Making Implementation Last: The Impact of Training Design on the Sustainability of an Evidence-Based Treatment in a Randomized Controlled Trial. Adm Policy Ment Health. 2021; 48(5):757-767. DOI: 10.1007/s10488-021-01126-6. View

13.

Faul F, Erdfelder E, Buchner A, Lang A . Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods. 2009; 41(4):1149-60. DOI: 10.3758/BRM.41.4.1149. View

14.

Brown C, Chamberlain P, Saldana L, Padgett C, Wang W, Cruden G . Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial. Implement Sci. 2014; 9:134. PMC: 4201704. DOI: 10.1186/s13012-014-0134-8. View

15.

Kolko D, Baumann B, Herschell A, Hart J, Holden E, Wisniewski S . Implementation of AF-CBT by community practitioners serving child welfare and mental health: a randomized trial. Child Maltreat. 2012; 17(1):32-46. DOI: 10.1177/1077559511427346. View

16.

Neta G, Clyne M, Chambers D . Dissemination and Implementation Research at the National Cancer Institute: A Review of Funded Studies (2006-2019) and Opportunities to Advance the Field. Cancer Epidemiol Biomarkers Prev. 2020; 30(2):260-267. DOI: 10.1158/1055-9965.EPI-20-0795. View

17.

Isaakidis P, Ioannidis J . Evaluation of cluster randomized controlled trials in sub-Saharan Africa. Am J Epidemiol. 2003; 158(9):921-6. DOI: 10.1093/aje/kwg232. View

18.

Wells K, Jones L, Chung B, Dixon E, Tang L, Gilmore J . Community-partnered cluster-randomized comparative effectiveness trial of community engagement and planning or resources for services to address depression disparities. J Gen Intern Med. 2013; 28(10):1268-78. PMC: 3785665. DOI: 10.1007/s11606-013-2484-3. View

19.

Lewis C, Fischer S, Weiner B, Stanick C, Kim M, Martinez R . Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015; 10:155. PMC: 4634818. DOI: 10.1186/s13012-015-0342-x. View

20.

Proctor E, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B . Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2008; 36(1):24-34. PMC: 3808121. DOI: 10.1007/s10488-008-0197-4. View