Article: A review of empirical research related to the use of small quantitative samples in clinical outcome scale development

Introduction: There has been a notable increase in the advocacy of using small-sample designs as an initial quantitative assessment of item and scale performance during the scale development process. This is particularly true in the development of clinical outcome assessments (COAs), where Rasch analysis has been advanced as an appropriate statistical tool for evaluating the developing COAs using a small sample.

Methods: We review the benefits such methods are purported to offer from both a practical and statistical standpoint and detail several problematic areas, including both practical and statistical theory concerns, with respect to the use of quantitative methods, including Rasch-consistent methods, with small samples.

Conclusions: The feasibility of obtaining accurate information and the potential negative impacts of misusing large-sample statistical methods with small samples during COA development are discussed.

Citing Articles

Measuring a family sense of coherence: a rasch-based study extending dyadic data analyses.

Mollerberg M, Arestedt K, Hagell P, Melin J BMC Palliat Care. 2025; 24(1):8.

PMID: 39789527 PMC: 11715594. DOI: 10.1186/s12904-024-01639-5.

Development and Construct Validity of the Work Instability Scale for People With Common Mental Disorders in a Sample of Depressed and Anxious Workers: A Rasch Analysis.

Danielsson L, Fornazar R, Holmgren K, Lundgren Nilsson A, Hensing G Rehabil Process Outcome. 2021; 9:1179572720936664.

PMID: 34497467 PMC: 8282151. DOI: 10.1177/1179572720936664.

Development and preliminary validation of the Adolescent Psoriasis Quality of Life instrument: a disease-specific measure of quality of life in adolescents with psoriasis.

Randa H, Khoury L, Gronborg T, Lomholt J, Skov L, Zachariae R Br J Dermatol. 2019; 183(1):96-104.

PMID: 31743420 PMC: 7383696. DOI: 10.1111/bjd.18719.

Scale development with small samples: a new application of longitudinal item response theory.

Houts C, Morlock R, Blum S, Edwards M, Wirth R Qual Life Res. 2018; 27(7):1721-1734.

PMID: 29423756 DOI: 10.1007/s11136-018-1801-z.

Measuring what matters to rare disease patients - reflections on the work by the IRDiRC taskforce on patient-centered outcome measures.

Morel T, Cano S Orphanet J Rare Dis. 2017; 12(1):171.

PMID: 29096663 PMC: 5667521. DOI: 10.1186/s13023-017-0718-x.

References

1.

Petrillo J, Cano S, McLeod L, Coon C . Using classical test theory, item response theory, and Rasch measurement theory to evaluate patient-reported outcome measures: a comparison of worked examples. Value Health. 2015; 18(1):25-34. DOI: 10.1016/j.jval.2014.10.005. View

2.

Preinerstorfer D, Formann A . Parameter recovery and model selection in mixed Rasch models. Br J Math Stat Psychol. 2011; 65(2):251-62. DOI: 10.1111/j.2044-8317.2011.02020.x. View

3.

Stone M, Yumoto F . The effect of sample size for estimating Rasch/IRT parameters with dichotomous items. J Appl Meas. 2004; 5(1):48-61. View

4.

Anthoine E, Moret L, Regnault A, Sebille V, Hardouin J . Sample size used to validate a scale: a review of publications on newly-developed patient reported outcomes measures. Health Qual Life Outcomes. 2014; 12:176. PMC: 4275948. DOI: 10.1186/s12955-014-0176-2. View

5.

Patrick D, Burke L, Gwaltney C, Leidy N, Martin M, Molsen E . Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2--assessing respondent understanding. Value Health. 2011; 14(8):978-88. DOI: 10.1016/j.jval.2011.06.013. View

6.

Meyer J, Hailey E . A Study of Rasch, partial credit, and rating scale model parameter recovery in WINSTEPS and jMetrik. J Appl Meas. 2012; 13(3):248-58. View

7.

Chen W, Lenderking W, Jin Y, Wyrwich K, Gelhorn H, Revicki D . Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Qual Life Res. 2013; 23(2):485-93. DOI: 10.1007/s11136-013-0487-5. View

8.

Karabatsos G . A critique of Rasch residual fit statistics. J Appl Meas. 2002; 1(2):152-76. View

9.

Linacre J . Investigating rating scale category utility. J Outcome Meas. 1999; 3(2):103-22. View

10.

Gorecki C, Lamping D, Nixon J, Brown J, Cano S . Applying mixed methods to pretest the Pressure Ulcer Quality of Life (PU-QOL) instrument. Qual Life Res. 2011; 21(3):441-51. DOI: 10.1007/s11136-011-9980-x. View

11.

Smith R, Schumacker R, Bush M . Using item mean squares to evaluate fit to the Rasch model. J Outcome Meas. 1998; 2(1):66-78. View

12.

Choi S, Cook K, Dodd B . Parameter recovery for the partial credit model using MULTILOG. J Outcome Meas. 1997; 1(2):114-42. View

13.

French G, Dodd B . Parameter recovery for the rating scale model using PARSCALE. J Outcome Meas. 1999; 3(2):176-99. View

A Review of Empirical Research Related to the Use of Small Quantitative Samples in Clinical Outcome Scale Development