When Does Consensus Exist in Expert Panels?

Increasingly, expert panels are being used to determine whether or not a consensus exists about criteria of good practice. It is, however, unclear how sensitive the panels' conclusions are to changes in the definitions of agreement and disagreement used. To explore this, two expert panels were established to assess the appropriate indications for cholecystectomy. Analyses of the results showed that the level of agreement depended on whether or not the views of outliers were included or eliminated. Exclusion of outliers increased the proportion of appropriate indications from about 40 per cent to 60 per cent. In contrast, the proportion of indications felt to be inappropriate was dependent on how strict the definitions employed were. Given that the principal purpose of expert panels is to inform quality assurance activities, the higher levels of agreement and disagreement achieved by eliminating outliers and employing more relaxed definitions are to be favoured.