Media and medicine,

asharma's picture

By now it should be clear that few things set me off more than shoddy media coverage of medical issues, particularly when important caveats are either omitted or distorted. Today's Globe and Mail (our national newspaper!!!) had a sensational front-page report on a study presented at a recent meeting by one of my colleages, which tripped every outrage switch in my head.

Since the study has not yet appeared in print, there is not even an opportunity to see full results or be reassured by an appropriate peer review. Though these comments won't appear in print, I will feel better if given an opportunity to ventilate. So here goes!


Dear Globe and Mail editors:

Your front-page coverage of the Dr. Michael Kramers’ study of childhood IQ and pre-term delivery (June 24, 2009) illustrates several common fallacies in statistical reasoning.

Firstly, as Dr. Kramer explained, this was an observational study and not a randomized trial. In the latter, otherwise comparable prospective mothers would have been randomly assigned to either term or pre-term delivery, intended to even out any differences in the two groups of children and allow any observed IQ disparity to be attributed to their birth timing. In an observational study, this selection is not random and the two groups of children are not necessarily comparable. In fact, we know that they differed in one important respect, in that one of the groups was delivered early. This usually means that their mothers suffered from a medical condition like pre-eclampsia, hypertension, diabetes, placenta previa, etc, which are known to compromise placental circulation and effect fetal development.

Secondly, it is a truism that an observational study cannot distinguish between causality and association. So even if birth timing and later IQ are associated, it is not possible to infer a causal relationship. It’s the old chicken-egg dilemma. On the basis of the data, one could just as easily argue that ‘newborns with higher IQ decide to remain in the womb longer’, and no amount of wishful thinking can overcome this fundamental limitation.

You also appear to misunderstand another important principle. Any two groups of children are likely to differ in average IQ, just by random chance. The critical question is whether the observed difference is large enough to suggest an explanation other than random chance. Any legitimate statistical analysis must therefore attempt to fix a precise value on this probability. This is usually calculated as the p-value (the probabilty of observing a difference this large by chance) or a confidence interval, to indicate the range in which the true value is likely to be found. By reporting a single value without an estimate of the uncertainty, your report is incomplete and uninterpretable. For example, if Dr. Kramer reported a p-value of 5% for this effect, we would accept the finding as ‘statistically significant’, since a difference of this size would be expected by chance alone in only 1/20 studies. This particular cut-off value is the standard one, but it is obviously not definitive, which is why medical studies showing weak effects have to be replicated before their conclusions can be accepted.

Lastly, even if a result is ‘statistically significant’, it is important not to confuse this with biological significance. Even if this were a randomized trial and the reported difference a real one, its size is so small as to be meaningless. Given the imprecision of childhood IQ tests in general, your reporter might well have asked after the likely real-world impact of a 1.7 point difference in IQ, in terms of either academic achievement or life skills. Had she done so, she would have discovered that this is not a meaningful difference, likely comparable to the impact of being exposed to shoddy journalism in the womb.

As pediatricians, we appreciate the genuine fears and anxieties faced by prospective parents. Since there are enough real dangers in the world, we would appreciate it if your paper didn’t manufacture imaginary ones through misreporting.

Celia Rodd MD, FRCP(C)
Atul Sharma MD, FRCP(C)
Department of Pediatrics, McGill University

Comments's picture

child IQ, gestational age, and breast feeding

The newspaper article mentions that this was an abstract presentation, but I can't seem to track down the original or any evidence of a more detailed paper yet. Do you know where it appeared? Or whether they adjusted for the covariates/confounders? (I know, I know - since we've had this exchange in this forum before - you're going to say that 'some of these confounders are on the causal pathway between exposure and effect', and you can't adjust for them because of Simpson's Paradox. I conceed. I was just curious :).

You didn't suggest otherwise, but your comments on 'observational studies' only apply to this analysis and not to their original study, which was a properly randomized, controlled, and peer-reviewed look at breast feeding on childhood IQ in Belarus, which showed a 4.7 point advantage. While I agree about inferring causality from observational data, that isn't an issue with the primary study. Just to be clear.

Your comments relate to a general debate on 'pre-publication by press conference'. As you pointed out, Dr. Kramer tried to explain the difference between randomized and observational studies, which Ms. McIlroy (the science reporter) didn't appear to grasp. How to improve this sort of reporting is a Gordian knot (more like an Augean stable) sort of problem. I thought they taught 'introductory statistics' in journalism school. Just think though, if this is what they do with subjects you understand, how can you trust their analysis of foreign policy or arms control? That's scary!! Since we have so little control over how things get covered by mass media, perhaps we need to return to the time when better journals returned submissions that had already been "published" in the lay press? That way, at least, you would always have the option of going to the source to clear up any confusion.


Creative Commons License | Powered by Drupal | Latest updates via RSS