Media Coverage of Antioxidant Study - Lies, Damned Lies and Statistics!

Some of the main points made by Drs. Hickey, Noriegai and Roberts are listed below.  

• The statistics were inappropriately applied to poorly selected data, thus the conclusions are invalid.

• Experimenter bias was compounded by a basic misuse of statistical testing.

• Of the initial studies, 14,910 (93%) were discarded, with only a brief explanation of the exclusion criteria.

• 747 (92%) of the 815 were rejected, for example, because no subject died during the experiment. The remaining 68 studies were included in the analysis.

• The large number of studies by Bjelakovic himself raises concerns in respect of objectivity, as the probability of trials being selected for inclusion in a meta-analysis can be influenced by knowledge of their results, leading to inclusion bias.

• A critical failing of the Bjelakovic paper is the absence of detail on the number of statistical tests performed on the data.

• The fact that this many tests were carried out on just one of the supplements (vitamin A) investigated suggests the results of the study are unreliable. Conventionally, a single statistical test has a 1 in 20 probability of being significant by chance alone. With 100 such tests, we would therefore expect five 'significant' results, just by chance.

• Media reports gave the impression that scientific evidence suggests vitamins may be harmful. In fact, no evidence has been provided to this effect. The statistics provided were insufficient to support a claim that vitamin supplements will increase mortality. Moreover, the results cannot validly be generalised to a relatively healthy general population

• Bjelakovic's meta-analysis has little biological meaning, because of the large number of ill-defined substances that have been grouped together. The meta-analysis includes a diverse range of doses of the individual supplements, with no concern for the expected physiological effects.

• The authors, by not controlling for experimenter bias, have produced a paper that might simply reflect their own personal bias.