Multiple Comparison Procedures - Cutting the Gordian Knot.

Multiple comparison procedures (MCPs), also known as mean separation tests, have been the subject of great controversy since the 1950s.  Essentially, these procedures are an attempt at simultaneously formulating and testing pair-wise comparison hypotheses using data from a single experiment.  Such attempts automatically cause alarm bells to go off in statisticians’ brains, and generate a desire for conservatism.  At the same time, statisticians recognize that the power of the procedure (the probability that it will detect real differences) is inversely related to the level of conservatism.  The conflict between these opposing desires causes individual statisticians to recommend multiple comparison procedures which span the full range from liberal to conservative.  This puts researchers in the unfortunate position of being caught in the cross-fire between statisticians of varying degrees of conservatism.

An unacceptable operating characteristic of most procedures is their “inconsistency”.  This characteristic led me to develop a “practical solution” to the MCP problem, which is to “cut the Gordian knot” by abandoning any attempt at simultaneous formulation and testing.  Instead, I recommend using the simplest multiple comparison procedure (the unrestricted LSD procedure) to formulate new hypotheses at a known “false discovery rate” (e.g., 5%), then independently test interesting new hypotheses in a second experiment.  Since this is normal scientific practice, this solution fits well with the way in which reputable scientists operate, and allows maximum probability of coming up with new ideas, in conjunction with a known type I error rate for the subsequent, independent testing of any interesting ideas thus generated. 

In the current talk, I discuss the above topics, illustrate the problem of inconsistency in relation to several procedures, and discuss the implications for sample size calculations of using multiple comparison procedures other than the simplest, unrestricted LSD procedure.