The Fundamentals of Multiplicity Adjustment in Biostatistics

The statistical problem of multiplicity is concerned with making protected multiple inferences and their valid interpretation in a particular study. Most discussions of multiplicity focus on the increase of type I error rate if testing is done without any adjustment, with only a few papers discussing its ramifications for type II errors/power. We provide a survey of main approaches to protected inference in biomedical studies, touching on procedures to control the family-wise error rate, false discovery rate, as well as false discovery exceedance probability. We discuss several notions of power including total power, average power, and power defined as exceedance probability for the true positive proportion. We provide commentary on best practices for adjusting for multiplicity in both type I and type II errors within families defined by primary, secondary, and exploratory endpoints in clinical trials and in experimental studies.