Having a Fit: Impact of Number of Items and Distribution of Data on Traditional Criteria for Assessing IRT's Unidimensionality Assumption

Overview

Journal Qual Life Res

Specialties Pharmacology
Rehabilitation Medicine

Date 2009 Mar 19

PMID 19294529

Citations 134

Authors

Karon F Cook

Michael A Kallen

Dagmar Amtmann

Affiliations

Soon will be listed here.

Abstract

Purpose: Confirmatory factor analysis fit criteria typically are used to evaluate the unidimensionality of item banks. This study explored the degree to which the values of these statistics are affected by two characteristics of item banks developed to measure health outcomes: large numbers of items and nonnormal data.

Methods: Analyses were conducted on simulated and observed data. Observed data were responses to the Patient-Reported Outcome Measurement Information System (PROMIS) Pain Impact Item Bank. Simulated data fit the graded response model and conformed to a normal distribution or mirrored the distribution of the observed data. Confirmatory factor analyses (CFA), parallel analysis, and bifactor analysis were conducted.

Results: CFA fit values were found to be sensitive to data distribution and number of items. In some instances impact of distribution and item number was quite large.

Conclusions: We concluded that using traditional cutoffs and standards for CFA fit statistics is not recommended for establishing unidimensionality of item banks. An investigative approach is favored over reliance on published criteria. We found bifactor analysis to be appealing in this regard because it allows evaluation of the relative impact of secondary dimensions. In addition to these methodological conclusions, we judged the items of the PROMIS Pain Impact bank to be sufficiently unidimensional for item response theory (IRT) modeling.

Citing Articles

Psychometric properties of the Arabic version of the Young Schema Questionnaire Short Form-Version 3 (YSQ-S3).

Sfeir M, Postigo A, Fekih-Romdhane F, Gonzalez-Nuevo C, Malaeb D, Hallit S Sci Rep. 2025; 15(1):406.

PMID: 39747149 PMC: 11697011. DOI: 10.1038/s41598-024-83089-w.

Four New Patient-Reported Outcome Measures Examining Health-Seeking Behavior in Persons With Type 2 Diabetes Mellitus (REDD-CAT): Instrument Development Study.

Mitchell S, Kallen M, Troost J, De La Cruz B, Bragg A, Martin-Howard J JMIR Diabetes. 2024; 9:e63434.

PMID: 39576685 PMC: 11624447. DOI: 10.2196/63434.

Initial development and validation of item banks to measure problematic hypersexuality.

van Tuijl P, Verboon P, van Lankveld J Open Res Eur. 2024; 3:129.

PMID: 39118807 PMC: 11306953. DOI: 10.12688/openreseurope.16131.2.

Using ECHO program data to develop a brief measure of caregiver support and cognitive stimulation using the home observation for measurement of the environment-infant/toddler (HOME-IT).

Trevino C, Lai J, Tang X, LeWinn K, Nozadi S, Wosu A Child Dev. 2024; 95(6):2241-2251.

PMID: 39080971 PMC: 11581929. DOI: 10.1111/cdev.14137.

Development and Psychometric Properties of Self-Reported Job Interview Skills and Job Interview Anxiety for Autistic Transition-Age Youth.

Genova H, Kallen M, Sherwood K, Dawalt L, Bishop L, Telfer D J Vocat Rehabil. 2024; 58(2):199-217.

PMID: 38974409 PMC: 11225930. DOI: 10.3233/jvr-230009.

References

Cook K, Teal C, Bjorner J, Cella D, Chang C, Crane P . IRT health outcomes data analysis project: an overview and summary. Qual Life Res. 2007; 16 Suppl 1:121-32. DOI: 10.1007/s11136-007-9177-5. View

Reise S, Waller N, Comrey A . Factor analysis and scale revision. Psychol Assess. 2000; 12(3):287-97. DOI: 10.1037//1040-3590.12.3.287. View

Hays R, Morales L, Reise S . Item response theory and health outcomes measurement in the 21st century. Med Care. 2000; 38(9 Suppl):II28-42. PMC: 1815384. DOI: 10.1097/00005650-200009002-00007. View

OConnor B . SPSS and SAS programs for determining the number of components using parallel analysis and velicer's MAP test. Behav Res Methods Instrum Comput. 2000; 32(3):396-402. DOI: 10.3758/bf03200807. View

Bentler P, Mooijaart A . Choice of structural model via parsimony: a rationale based on precision. Psychol Bull. 1989; 106(2):315-7. DOI: 10.1037/0033-2909.106.2.315. View

Lai J, Crane P, Cella D . Factor analysis techniques for assessing sufficient unidimensionality of cancer related fatigue. Qual Life Res. 2006; 15(7):1179-90. DOI: 10.1007/s11136-006-0060-6. View

Cook K, OMalley K, Roddey T . Dynamic assessment of health outcomes: time to let the CAT out of the bag?. Health Serv Res. 2005; 40(5 Pt 2):1694-711. PMC: 1361218. DOI: 10.1111/j.1475-6773.2005.00446.x. View

Reeve B, Burke L, Chiang Y, Clauser S, Colpe L, Elias J . Enhancing measurement in health outcomes research supported by Agencies within the US Department of Health and Human Services. Qual Life Res. 2007; 16 Suppl 1:175-86. DOI: 10.1007/s11136-007-9190-8. View

Bentler P . Comparative fit indexes in structural models. Psychol Bull. 1990; 107(2):238-46. DOI: 10.1037/0033-2909.107.2.238. View

10.

McDonald R, Mok M . Goodness of Fit in Item Response Models. Multivariate Behav Res. 2016; 30(1):23-40. DOI: 10.1207/s15327906mbr3001_2. View

11.

Reise S, Haviland M . Item response theory and the measurement of clinical change. J Pers Assess. 2005; 84(3):228-38. DOI: 10.1207/s15327752jpa8403_02. View

12.

Mannion A, Elfering A, Staerkle R, Junge A, Grob D, Semmer N . Outcome assessment in low back pain: how low can you go?. Eur Spine J. 2005; 14(10):1014-26. DOI: 10.1007/s00586-005-0911-9. View

13.

Bjorner J, Kosinski M, Ware Jr J . Calibration of an item pool for assessing the burden of headaches: an application of item response theory to the headache impact test (HIT). Qual Life Res. 2003; 12(8):913-33. DOI: 10.1023/a:1026163113446. View

14.

Browne M . Asymptotically distribution-free methods for the analysis of covariance structures. Br J Math Stat Psychol. 1984; 37 ( Pt 1):62-83. DOI: 10.1111/j.2044-8317.1984.tb00789.x. View

15.

DeWalt D, Rothrock N, Yount S, Stone A . Evaluation of item candidates: the PROMIS qualitative item review. Med Care. 2007; 45(5 Suppl 1):S12-21. PMC: 2810630. DOI: 10.1097/01.mlr.0000254567.79743.e2. View

16.

Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B . The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007; 45(5 Suppl 1):S3-S11. PMC: 2829758. DOI: 10.1097/01.mlr.0000258615.42478.55. View

17.

Reise S, Morizot J, Hays R . The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Qual Life Res. 2007; 16 Suppl 1:19-31. DOI: 10.1007/s11136-007-9183-7. View

18.

Hu L, Bentler P, Kano Y . Can test statistics in covariance structure analysis be trusted?. Psychol Bull. 1992; 112(2):351-62. DOI: 10.1037/0033-2909.112.2.351. View

19.

Reise S, Widaman K, Pugh R . Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. Psychol Bull. 1993; 114(3):552-566. DOI: 10.1037/0033-2909.114.3.552. View

20.

Reeve B, Hays R, Bjorner J, Cook K, Crane P, Teresi J . Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007; 45(5 Suppl 1):S22-31. DOI: 10.1097/01.mlr.0000250483.85507.04. View