I Think You'll Find It's a Bit More Complicated Than That (9 page)

Guardian
, 20 August 2011

What do all these numbers mean? ‘
“Worrying” Jobless Rise
Needs Urgent Action – Labour’ was the BBC headline. It explained the problem in its own words: ‘The number of people out of work rose by 38,000 to 2.49 million in the three months to June, official figures show.’

There are dozens of different ways to quantify the jobs market, and I’m not going to summarise them all here. The claimant count and the labour force survey are commonly used, and the number of hours worked is informative too: you can fight among yourselves over which is best, and get distracted by party politics to your hearts’ content. But in claiming that this figure for the number of people out of work has risen, the BBC is simply wrong.

Here’s why. The
‘Labour Market’ figures
come through the Office for National Statistics, and it has published the latest numbers
in a PDF document
.
See here
, top table, fourth row, you will find these figures the BBC is citing. Unemployment aged sixteen and above is at 2,494,000, and has risen by 38,000 over the past quarter (and by 32,000 over the past year). But you will also see some other figures, after the symbol ‘±’, in a column marked ‘sampling variability of change’.

Those figures are called ‘95 per cent confidence intervals’, and these are among the most useful inventions of modern life.

We can’t do a full census of everyone in the population every time we want some data, because they’re too expensive and time-consuming for monthly data collection. Instead, we take what we hope is a representative sample.

This can fail in two interesting ways. Firstly, you’ll be familiar with the idea that a sample can be
systematically
unrepresentative: if you want to know about the health of the population as a whole, but you survey people in a GP waiting room, then you’re an idiot.

But a sample can also be unrepresentative simply by chance, through something called sampling error. This is not caused by idiocy. Imagine a large bubblegum-vending machine, containing thousands of blue and yellow bubblegum balls. You know that exactly 40 per cent of those balls are yellow. When you take a sample of a hundred balls, you might get forty yellow ones, but in fact, as you intuitively know already, sometimes you will get thirty-two, sometimes forty-eight, or thirty-seven, or forty-three, or whatever. This is sampling error.

Now, normally, you’re at the other end of the telescope. You take your sample of a hundred balls, but you don’t know the true proportion of yellow balls in the jar – you’re trying to estimate that – so you calculate a 95 per cent confidence interval around whatever proportion of yellow you get in your sample of a hundred balls, using a formula (in this case, 1.96 × the square root of ((0.6 × 0.4) ÷ 100)).

What does this mean? Strictly (it still makes my head hurt), it means that if you repeatedly took samples of a hundred, then on 95 per cent of those attempts, the true proportion in the bubblegum jar would lie somewhere between the upper and lower limits of the 95 per cent confidence intervals of your samples. That’s all we can say.

So, if we look at these employment figures, you can see that the changes reported are clearly not statistically significant: the estimated change over the past quarter is 38,000, but the 95 per cent confidence interval is ±87,000, running from –49,000 to 125,000. That wide range clearly includes zero, which means it’s perfectly likely that there’s been no change at all. The annual change is 32,000, but again, that’s ±111,000.

I don’t know what’s happening to the economy – it’s probably not great. But these specific numbers are being over-interpreted, and there is an equally important problem arising from that, which is frankly more enduring for meaningful political engagement.

We are barraged, every day, with a vast quantity of numerical data, presented with absolute certainty and fetishistic precision. In reality, many of these numbers amount to nothing more than statistical noise, the gentle static fuzz of random variation and sampling error, making figures drift up and down, following no pattern at all, like the changing roll of a dice. This, I confidently predict, will never change.

Scientific Proof
That We Live in a Warmer and More Caring Universe

Guardian
, 29 November 2008

As usual, it’s not Watergate, it’s just slightly irritating.
‘Down’s births increase in a caring Britain’
, said
The Times
: ‘More babies are being born with Down’s syndrome as parents feel increasingly that society is a more welcoming place for children with the condition.’ That’s beautiful. ‘More mothers are choosing to keep their babies
when diagnosed with Down’s
Syndrome’ said the
Mail
. ‘Parents appear to be more willing to bring a child with Down’s syndrome into the world because British society has become increasingly accepting of the genetic abnormality’
said the
Independent
. “Children’s quality of life is better and acceptance has risen’,
said the
Mirror
.

Their quoted source was no less impeccable than a BBC
Radio 4 documentary
presented by Felicity Finch (
her what plays Ruth Archer
), broadcast on Monday. ‘The number of babies with Down syndrome has steadily fallen, that is until today, when for the first time ever that number is higher than before, when testing was introduced.’ I see. ‘I’m keen to find out why more parents are making this decision.’ They’re not. ‘I was so intrigued by these figures that I’ve been following some parents to find out what lies behind their choice.’ Felicity, they’re not. The entire founding premise of your entire twenty-seven-minute documentary is wrong.

There has indeed been a 4 per cent increase in Down’s syndrome live births in England and Wales from 1989 to 2006 (717 and 749 affected births in the two years, respectively). However, since 1989 there has also been a far greater increase in the number of Down’s syndrome foetuses created in the first place, because people are getting pregnant much later in life.

What causes Down’s syndrome? We don’t really know, but maternal age is the only well-recognised association. Your risk of a Down’s syndrome pregnancy below the age of twenty-five is about one in 1,600. This rises to about one in 340 at thirty-five, and one in forty at the age of forty-three. In 1989, 6 per cent of pregnant women were over thirty-five years of age. By 2006 it was 15 per cent.

The
National Down Syndrome Cytogenetic Register
holds probably the largest single dataset on Down’s syndrome, with over 17,000 anonymous records collected since 1989, making it one of the most reliable resources in the search for patterns and possible causal factors. They have calculated that if you account for the increase in the age at which women are becoming pregnant, from 1989 to 2006 the number of Down’s syndrome live births in the UK would have increased not by 4 per cent, but from 717 to an estimated 1,454, if screening and subsequent termination had not been available.

Except, of course, antenatal screening is widely available, it is widely taken up, and contrary to what every newspaper told you this week, it is widely acted upon. More than nine out of ten women who have an antenatal diagnosis of Down’s syndrome decide to have a termination of the pregnancy. This proportion has not changed since 1989. This is the ‘decision’ that Felicity Finch, Radio 4, the
Mail
,
The Times
, the
Mirror
and the rest are claiming more parents are taking: to carry on with a Down’s syndrome pregnancy. This is what they are taking as evidence of a more caring society. But the figure has not changed.

Since we’ve now established beyond any doubt that the team behind this documentary got their numbers – and therefore their whole factual premise – entirely wrong, I think we’re also entitled to engage with their crass moral judgements. If I terminate a Down’s syndrome pregnancy, is that proof that society is not a warm, caring place, and that I am not a warm, caring person? For many parents the decision to terminate will be a difficult and upsetting one, especially later in life, and stories like this create a pretty challenging backdrop for making it. This would have been true even if the programme-makers had got their figures absolutely perfect, but as is so often the case for those with spare flesh to wave at strangers, their facts and figures are simply wrong.

The
National Down Syndrome Cytogenetic Register
felt obliged to issue a thorough clarification. The thoroughly brilliant ‘Behind the Headlines’ service on the NHS Choices website
took the story to pieces
, as it so often does, in its daily round-up of the real evidence behind the health news (disclosure: I had a trivially tiny hand in helping to set this service up).

Everybody ignored them, nobody has clarified, and
Born With Down’s
remains ‘Choice of the Day’ on the
Radio 4 website
.

Drink Coffee
, See Dead People

Guardian
, 17 January 2009

‘Danger from just
7 cups of coffee a day
’, said the
Express
on Wednesday. ‘Too much coffee can make you hallucinate and sense dead people say sleep experts. The equivalent of just seven cups of instant coffee a day is enough to trigger the weird responses.’ The story appeared
in almost every national newspaper
.

This was weak observational data. That’s just the start of our story, but you should know
exactly what the researchers did
. They sent an email inviting students to fill out an online survey, and 219 agreed.

The
survey is still online
(I just clicked answers randomly to see the next question until I got to the end). It asks about caffeine intake in vast detail, and then uses one scale to measure how prone you are to feeling persecuted, and another, the
‘Launay-Slade Hallucination Scale’
, sixteen questions designed to measure ‘predisposition to hallucination-like experiences’.

Some of these questions are about having hallucinations and seeing ghosts, but some really are a very long way from there. Heavy coffee drinkers could have got higher scores on this scale by responding affirmatively to statements like: ‘No matter how hard I try to concentrate on my work, unrelated thoughts always creep into my mind’; ‘Sometimes a passing thought will seem so real that it frightens me’; or ‘Sometimes my thoughts seem as real as actual events in my life.’ That’s not seeing ghosts or hearing voices.

And of course, this was weak observational data, and there could have been
alternative explanations
for the observed correlation between caffeine intake and very slightly higher LSHS scores. Maybe some students who drink a lot of coffee are also sleep-deprived, and marginally more prone to hallucinations because of that. Maybe they are drinking coffee to help them get over last night’s massive marijuana hangover.

Maybe the kinds of people who take drugs instrumentally to have fun and distort their perceptions also take drugs like caffeine instrumentally to stay alert. You can think of more, I’m sure. The researchers were keen to point out this shortcoming in their paper. The
Express
and many others didn’t seem to care.

Then, if you read
the academic paper
, you find that the associations reported are weak. For the benefit of those who understand ‘regression’ (and it makes anybody’s head hurt), 18 per cent of the variance in the LSHS score is explained by gender, age and stress. When you add in caffeine to those three things, 21 per cent of the variance in the LSHS score is explained: only an extra 3 per cent, so caffeine adds very little. The finding is statistically significant, as the researchers point out, so it’s unlikely to be due to chance, but that doesn’t change the fact that it’s still weak, and it still explains only a tiny amount of the overall variance in scores on the ‘predisposed-to-hallucinations’ scale.

Lastly, most newspapers reported a rather dramatic claim, that seven cups of coffee a day is associated with a three times higher prevalence of hallucinations. This figure does not appear anywhere in the paper. It seems to be an ad hoc analysis done afterwards by the researchers, and put into
the press release
, so you cannot tell how they did it, or whether they controlled appropriately for problems in the data, like something called
‘multiple comparisons’
.

Here is the problem. Apparently this three-times-greater risk is for the top 10 per cent of caffeine consumers, compared with the bottom 10 per cent. They say that heavy caffeine drinkers were three times more likely to have answered affirmatively to just one LSHS question: ‘In the past, I have had the experience of hearing a person’s voice and then found that
no one was there
.’

Now, this poses massive problems. Imagine that I am stood facing a barn, holding a machine gun, blindfolded, firing off shots whilst swinging my whole body from side to side and laughing maniacally. I then walk up to the barn, find three bullet holes which happen to be very close together, and
draw a target around them
, claiming I am an excellent shot.

You can easily find patterns in your data once it’s collected. Why choose 10 per cent as your cut-off? Why not the top and bottom quarters? Maybe they have accounted for this problem. You don’t know. I don’t know. They say they have, to me, in emails, but it wasn’t in the paper, and we can’t all see the details. I don’t think that’s satisfactory for a headline finding, and the first claim of a press release.

Then there is one final problem: putting a finding in the press release but not into the paper is a subversion of the peer-review process. People will read this coverage, they will be scared, and they will change their behaviour. But the researchers’ key reported claim, with massive popular impact, was never peer reviewed, and crucially the technical details behind it are not in the public domain.