An Empirical Study of the Possible Relation of Treatment Differences to Quality Scores in Controlled Randomized Clinical Trials

Meta-analytic investigations sometimes use assessments of research quality according to a formal protocol as a tool for improving research synthesis. We asked whether a particular quality scoring system could have a direct use in adjusting the summary estimates of a treatment difference. In an empirical study of the relation of quality scores to treatment differences in published meta-analyses of 7 groups of controlled randomized clinical trials comprising 107 primary studies, we found no relation between treatment difference and overall quality score. We also found no relation between quality score and variation in treatment difference. The level of quality scores has increased at a rate of 9% per decade for three decades, averaging 0.51 on a scale of 0 to 1 for the 1980s, and leaving much room for improvement. Nevertheless, attention to quality of studies by editors, reviewers, and authors may be raising both the level of research done and quality of the reports.