A Proposal for the Assessment of Replication of Effects in Single-case Experimental Designs

Overview

Journal J Appl Behav Anal

Specialty Social Sciences

Date 2022 Apr 25

PMID 35467023

Authors

Rumen Manolov

Rene Tanious

Belen Fernandez-Castilla

Affiliations

Soon will be listed here.

Abstract

In science in general and in the context of single-case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been replicated or not (or to what extent) are scarce, in the general context of behavioral sciences, and practically null in the single-case experimental designs context. We propose an extension of the modified Brinley plot for assessing how many of the effects replicate. To make this assessment possible, a definition of replication is suggested, on the basis of expert judgment, rather than on statistical criteria. The definition of replication and its graphical representation are justified, presenting their strengths and limitations, and illustrated with real data. A user-friendly software is made available for obtaining automatically the graphical representation.

References

Perone M . Statistical inference in behavior analysis: Experimental control is better. Behav Anal. 2012; 22(2):109-16. PMC: 2731354. DOI: 10.1007/BF03391988. View

Parker R, Vannest K, Davis J . Effect size in single-case research: a review of nine nonoverlap techniques. Behav Modif. 2011; 35(4):303-22. DOI: 10.1177/0145445511399147. View

Manolov R, Tanious R, Onghena P . Quantitative Techniques and Graphical Representations for Interpreting Results from Alternating Treatment Design. Perspect Behav Sci. 2022; 45(1):259-294. PMC: 8894511. DOI: 10.1007/s40614-021-00289-9. View

Hantula D . Editorial: Replication and Reliability in Behavior Science and Behavior Analysis: A Call for a Conversation. Perspect Behav Sci. 2020; 42(1):1-11. PMC: 6701722. DOI: 10.1007/s40614-019-00194-2. View

Moeyaert M, Yang P, Xu X, Kim E . Characteristics of Moderators in Meta-Analyses of Single-Case Experimental Design Studies. Behav Modif. 2021; 47(6):1510-1545. DOI: 10.1177/01454455211002111. View

Anzures-Cabrera J, Higgins J . Graphical displays for meta-analysis: An overview with suggestions for practice. Res Synth Methods. 2015; 1(1):66-80. DOI: 10.1002/jrsm.6. View

Tate R, Perdices M, Rosenkoetter U, Wakim D, Godbee K, Togher L . Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: the 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychol Rehabil. 2013; 23(5):619-38. DOI: 10.1080/09602011.2013.824383. View

Fahmie T, Hanley G . Progressing toward data intimacy: a review of within-session data analysis. J Appl Behav Anal. 2008; 41(3):319-31. PMC: 2521866. DOI: 10.1901/jaba.2008.41-319. View

Caneiro J, Smith A, Linton S, Moseley G, OSullivan P . How does change unfold? an evaluation of the process of change in four people with chronic low back pain and high pain-related fear managed with Cognitive Functional Therapy: A replicated single-case experimental design study. Behav Res Ther. 2019; 117:28-39. DOI: 10.1016/j.brat.2019.02.007. View

10.

Sanabria F, Killeen P . BETTER STATISTICS FOR BETTER DECISIONS: REJECTING NULL HYPOTHESES STATISTICAL TESTS IN FAVOR OF REPLICATION STATISTICS. Psychol Sch. 2009; 44(5):471-481. PMC: 2613360. DOI: 10.1002/pits.20239. View

11.

Harrington M, Velicer W . Comparing Visual and Statistical Analysis in Single-Case Studies Using Published Studies. Multivariate Behav Res. 2015; 50(2):162-83. PMC: 4677800. DOI: 10.1080/00273171.2014.973989. View

12.

Snodgrass M, Meadan H, Chung M, Biggs E . Graphing the Intersection of Rate and Fidelity in Single-Case Research. Behav Anal Pract. 2022; 15(1):284-294. PMC: 8854548. DOI: 10.1007/s40617-021-00556-w. View

13.

Jacobson N, Truax P . Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991; 59(1):12-9. DOI: 10.1037//0022-006x.59.1.12. View

14.

Onghena P, Edgington E . Customization of pain treatments: single-case design and analysis. Clin J Pain. 2004; 21(1):56-68. DOI: 10.1097/00002508-200501000-00007. View

15.

Michiels B, Heyvaert M, Meulders A, Onghena P . Confidence intervals for single-case effect size measures based on randomization test inversion. Behav Res Methods. 2016; 49(1):363-381. DOI: 10.3758/s13428-016-0714-4. View

16.

Porcino A, Shamseer L, Chan A, Kravitz R, Orkin A, Punja S . SPIRIT extension and elaboration for n-of-1 trials: SPENT 2019 checklist. BMJ. 2020; 368:m122. DOI: 10.1136/bmj.m122. View

17.

Walker S, Carr J . Generality of Findings From Single-Case Designs: It's Not All About the "". Behav Anal Pract. 2021; 14(4):991-995. PMC: 8586328. DOI: 10.1007/s40617-020-00547-3. View

18.

Dixon P, Glover S . Assessing evidence for replication: A likelihood-based approach. Behav Res Methods. 2020; 52(6):2452-2459. DOI: 10.3758/s13428-020-01403-6. View

19.

Hillary F, Medaglia J . What the replication crisis means for intervention science. Int J Psychophysiol. 2019; 154:3-5. PMC: 6842660. DOI: 10.1016/j.ijpsycho.2019.05.006. View

20.

Durlak J . How to select, calculate, and interpret effect sizes. J Pediatr Psychol. 2009; 34(9):917-28. DOI: 10.1093/jpepsy/jsp004. View