Healthy Skepticism Library item: 12477

Warning: This library includes all items relevant to health product marketing that we are aware of regardless of quality. Often we do not agree with all or part of the contents.

Publication type: Journal Article

Vickers AJ.
Stats for the Health Professional - Statistics Is Unscientific! (Well, as Clinicians See it, Anyway)
Medscape Business of Medicine 2008 Jan 7;

Full text:

I once read a letter to the editor in which the correspondent, a clinician, described a paper’s statistical analysis as “not strictly kosher.” This might well be true, on the grounds that the statistician involved had probably not been certified by the appropriate rabbinic authorities. Leaving aside the issue of whether the Mann-Whitney test counts as milk or meat, the idea of “kosher statistics” does give a wonderful insight in to how many clinicians view statistics.
Statistics is often seen as a set of laws, handed down from above, violation of which constitutes a transgression. As a statistician, I am repeatedly asked whether a particular statistical analysis is “allowed” or whether it would be “against the rules”; as a statistics teacher, my students’ questions often concern “right and wrong.”

It is hard to think of any other area of science that is characterized by so many religious and legalistic metaphors. We don’t wonder whether, say, use of frozen and rethawed serum for a biomarker analysis would break any laws, or whether errors in flow cytometry technique constitute an eternal or just a venial sin. Hence my view that statistics, as clinicians see it, is unscientific.

One of science’s defining characteristics is that new ideas are developed, and that both new and old ideas are tested empirically. This is as true for statistics as for any other science. Many of the techniques I use in my day-to-day work
— Cox regression, bootstrapping, k-fold cross validation, general estimating equations — were invented relatively recently. I myself have developed a new statistical technique, decision curve analysis (which I’d be happy to explain to any reader experiencing insomnia). Moreover, statisticians test statistical methods experimentally: we have computers simulate data sets, then apply different statistical methods and see which come up with the right answers. If I recall correctly, the clinician who liked his statistics kosher was concerned about the use of a t-test on skewed data.
Statisticians have tried applying the t-test to data simulated with skew and have found out that, although it sometimes fails to detect a true difference between groups, the t-test works just fine in terms of not telling you that there is a difference when there isn’t. The trial that this clinician criticized reported that the drug worked; the use of the t-test was therefore not a problem.

So, no, the t-test was not inscribed on the stone tablets Moses brought down from Mount Sinai along with the commandment “Thou shalt not use with skewed data.” The t-test was invented by a statistician (a guy who worked in a beer company) and has been subsequently been tested by other statisticians (including me) to find out if it is any good (it turns out it isn’t ideal for the applications I need, and I rarely use it at all). Just like any other science, what you want to know about any statistical technique is the degree to which it might give you a wrong answer, and whether there other methods around that give you a better chance of getting things right. There aren’t rules, laws, and commandments about this; you just have to know the latest research data.

Note: Andrew Vickers is Jewish, but admits that he has not had his statistical software certified as glatt.

Andrew J. Vickers, PhD, Assistant Attending Research Methodologist, Memorial Sloan-Kettering Cancer Center, New York, NY

Andrew J. Vickers, PhD, has disclosed no relevant financial relationships.