A General Critical Appraisal Tool: an Evaluation of Construct Validity

Background: Many critical appraisal tools (CATs) exist for which there is little or no information on development of the CAT, evaluation of validity, or testing reliability. The proposed CAT was developed based on a number of other CATs, general research methods theory, and reporting guidelines but requires further study to determine its effectiveness.

Objectives: To establish a scoring system and to evaluate the construct validity of the proposed critical appraisal tool before undertaking reliability testing.

Methods: Data obtained from this exploratory study along with information on the design of the proposed CAT were combined to evaluate construct validity using the Standards for educational and psychological testing which consist of five types of evidence: test content, response process, internal structure, relations to other variables, and consequences of testing. To obtain data for internal structure and relations to other variables, the proposed CAT was analysed against five alternative CATs. A random sample of 10 papers from six different research designs across the range of health related research were selected, giving a total sample size of 60 papers.

Results: In all research designs, the proposed CAT had significant (p<0.05, two-tailed) weak to moderate positive correlations (Kendall's τ 0.33-0.55) with the alternative CATs, except in the Preamble category. There were significant moderate to strong positive correlations in the quasi-experimental (τ 0.70-1.00), descriptive/exploratory/observational (τ 0.72-1.00), qualitative (τ 0.74-0.81), and systematic review (τ 0.62-0.82) designs and to a lesser extent in the true experimental (τ 0.68-0.70) design. There were no significant correlations in the single system research designs.

Conclusions: Based on the results obtained, the theory on which the proposed CAT was designed, and the objective of the proposed CAT there was enough evidence to show that inferences made from scores obtained from the proposed CAT should be sound.