Confidence in Mathematics is Confounded by Responses to Reverse-coded Items

Overview

Journal Front Psychol

Date 2024 Nov 8

PMID 39512574

Authors

Faye Antoniou

Mohammed H Alghamdi

Affiliations

Soon will be listed here.

Abstract

Introduction: This study investigates the confounding effects of reverse-coded items on the measurement of confidence in mathematics using data from the 2019 Trends in International Mathematics and Science Study (TIMSS).

Methods: The sample came from the Saudi Arabian cohort of 8th graders in 2019 involving 4,515 students. Through mixture modeling, two subgroups responding in similar ways to reverse-coded items were identified representing approximately 9% of the sample.

Results: Their response to positively valenced and negatively valenced items showed inconsistency and the observed unexpected response patterns were further verified using Lz*, U3, and the number of Guttman errors person fit indicators. Psychometric analyses on the full sample and the truncated sample after deleting the aberrant responders indicated significant improvements in both internal consistency reliability and factorial validity.

Discussion: It was concluded that reverse-coded items contribute to systematic measurement error that is associated with distorted item level parameters that compromised the scale's reliability and validity. The study underscores the need for reconsideration of reverse-coded items in survey design, particularly in contexts involving younger populations and low-achieving students.

References

Jaensson M, Nilsson U . Impact of changing positively worded items to negatively worded items in the Swedish web-version of the Quality of Recovery (SwQoR) questionnaire. J Eval Clin Pract. 2016; 23(3):502-507. DOI: 10.1111/jep.12639. View

Marsh H . Positive and negative global self-esteem: a substantively meaningful distinction or artifactors?. J Pers Soc Psychol. 1996; 70(4):810-9. DOI: 10.1037//0022-3514.70.4.810. View

Kam C . Why Do Regular and Reversed Items Load on Separate Factors? Response Difficulty vs. Item Extremity. Educ Psychol Meas. 2023; 83(6):1085-1112. PMC: 10638982. DOI: 10.1177/00131644221143972. View

Kam C, Meyer J . Testing the Nonlinearity Assumption Underlying the Use of Reverse-Keyed Items: A Logical Response Perspective. Assessment. 2022; 30(5):1569-1589. DOI: 10.1177/10731911221106775. View

Dominguez-Salas S, Andres-Villas M, Riera-Sampol A, Tauler P, Bennasar-Veny M, Aguilo A . Analysis of the psychometric properties of the Sense of Coherence scale (SOC-13) in patients with cardiovascular risk factors: a study of the method effects associated with negatively worded items. Health Qual Life Outcomes. 2022; 20(1):8. PMC: 8751372. DOI: 10.1186/s12955-021-01914-6. View

Pedersen H, Sparle Christensen K, Prior A, Christensen K . The dimensionality of the Perceived Stress Scale: The presence of opposing items is a source of measurement error. J Affect Disord. 2023; 344:485-494. DOI: 10.1016/j.jad.2023.10.109. View

Suarez-Alvarez J, Pedrosa I, Lozano L, Garcia-Cueto E, Cuesta M, Muniz J . Using reversed items in Likert scales: A questionable practice. Psicothema. 2018; 30(2):149-158. DOI: 10.7334/psicothema2018.33. View

Koutsogiorgi C, Michaelides M . Response tendencies due to item wording using eye-tracking methodology accounting for individual differences and item characteristics. Behav Res Methods. 2022; 54(5):2252-2270. DOI: 10.3758/s13428-021-01719-x. View

Beck M, Albano A, Smith W . Person-Fit as an Index of Inattentive Responding: A Comparison of Methods Using Polytomous Survey Data. Appl Psychol Meas. 2019; 43(5):374-387. PMC: 6572906. DOI: 10.1177/0146621618798666. View

10.

Podsakoff P, Mackenzie S, Lee J, Podsakoff N . Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003; 88(5):879-903. DOI: 10.1037/0021-9010.88.5.879. View

11.

Lakens D . Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013; 4:863. PMC: 3840331. DOI: 10.3389/fpsyg.2013.00863. View

12.

Plieninger H, Heck D . A New Model for Acquiescence at the Interface of Psychometrics and Cognitive Psychology. Multivariate Behav Res. 2018; 53(5):633-654. DOI: 10.1080/00273171.2018.1469966. View

13.

Reise S, Kim D, Mansolf M, Widaman K . Is the Bifactor Model a Better Model or Is It Just Better at Modeling Implausible Responses? Application of Iteratively Reweighted Least Squares to the Rosenberg Self-Esteem Scale. Multivariate Behav Res. 2016; 51(6):818-838. PMC: 5312782. DOI: 10.1080/00273171.2016.1243461. View

14.

Garcia-Pardina A, Abad F, Christensen A, Golino H, Garrido L . Dimensionality assessment in the presence of wording effects: A network psychometric and factorial approach. Behav Res Methods. 2024; 56(6):6179-6197. DOI: 10.3758/s13428-024-02348-w. View

15.

Steinmann I, Strietholt R, Braeken J . A constrained factor mixture analysis model for consistent and inconsistent respondents to mixed-worded scales. Psychol Methods. 2021; 27(4):667-702. DOI: 10.1037/met0000392. View

16.

Kam C, Meyer J, Sun S . Why Do People Agree With Both Regular and Reversed Items? A Logical Response Perspective. Assessment. 2021; 28(4):1110-1124. DOI: 10.1177/10731911211001931. View

17.

Clauss K, Bardeen J . Addressing Psychometric Limitations of the Attentional Control Scale via Bifactor Modeling and Item Modification. J Pers Assess. 2018; 102(3):415-427. DOI: 10.1080/00223891.2018.1521417. View

18.

Kam C . Novel Insights Into Item Keying/Valence Effect Using Latent Difference (LD) Modeling Analysis. J Pers Assess. 2017; 100(4):389-397. DOI: 10.1080/00223891.2017.1369095. View

19.

Schroeders U, Schmidt C, Gnambs T . Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting. Educ Psychol Meas. 2022; 82(1):29-56. PMC: 8725053. DOI: 10.1177/00131644211004708. View

20.

Marsh H, Scalas L, Nagengast B . Longitudinal tests of competing factor structures for the Rosenberg Self-Esteem Scale: traits, ephemeral artifacts, and stable response styles. Psychol Assess. 2010; 22(2):366-81. DOI: 10.1037/a0019225. View