Trick or Treatment (20 page)

On its own, the third experiment seems to suggest that astrology works, because a hit rate equivalent to 5 out of 20 is much higher than chance would predict. Indeed, the majority of experiments (three out of five) imply a higher than expected hit rate, so one way to interpret these sets of data would be to conclude that, in general, the experiments support astrology. However, a meta-analysis would come to a different conclusion.

The meta-analysis would start by pointing out that the number of attempts made by the astrologer in any one of the experiments was relatively small, and therefore the result of any single experiment could be explained by mere chance. In other words, the result of any one of these experiments is effectively meaningless. Next, the researcher doing the meta-analysis would combine all the data from the individual experiments as though they were part of one giant experiment. This tells us that the astrologer had 49 hits out of 600 in total, which is equivalent to a hit rate of 0.98 out of 12, which is very close to 1 out of 12, the hit rate expected by chance alone. The conclusion of this hypothetical meta-analysis would be that the astrologer has demonstrated no special ability to determine a person’s star sign based on their personality. This conclusion is far more reliable than anything that could have been deduced solely from any one of the small-scale experiments. In scientific terms: a meta-analysis is said to minimize random and selection biases.

Turning to medical research, there are numerous treatments that have been tested by meta-analysis. For example, in the 1980s researchers wanted to know if corticosteroid medication could help reduce respiratory problems in premature babies. They designed a trial which involved giving the treatment to pregnant women likely to have premature births and then monitoring the babies born to these mothers. Ideally, the researchers would have conducted one trial in a single hospital with a large number of cases, but it was only possible to identify a few suitable cases each year per hospital, so it would have taken several years to accumulate sufficient data in this manner. Instead, the researchers conducted several trials across several hospitals. The results of each individual trial varied from hospital to hospital, because the numbers of babies in each trial was small and random influences were large. Yet a meta-analysis of all the trials showed with certainty that corticosteroid medication during pregnancy benefited premature babies. This treatment is part of the reason why the number of infant deaths due to respiratory distress syndrome has fallen dramatically – there were 25,000 such deaths in America in the early 1950s and today the number is fewer than 500.

The meta-analysis in the premature baby study was fairly straightforward, because the individual trials were similar to each other and so they could be merged easily. The same is true of the hypothetical example concerning astrology. Unfortunately, conducting a meta-analysis is often a messy business, because the individual trials have generally been conducted in different ways. Trials for the same medication might vary according to the dose given, the period of monitoring, and so on. In Linde’s case, the meta-analysis was particularly problematic. In order to draw a conclusion about the efficacy of homeopathy, Linde was attempting to include homeopathy trials investigating a huge variety of remedies, across a range of potencies, being used to treat a wide range of conditions, such as asthma and minor burns.

Linde trawled through the computer databases, attended numerous homeopathic conferences, contacted researchers in the field and eventually found 186 published trials on homeopathy. He and his colleagues then decided to exclude from his meta-analysis those trials that failed to meet certain basic conditions. For example, in addition to a group of patients being treated with homeopathy and a control group of patients, an acceptable trial had to have a placebo for the control-group patients, or there had to be random allocation of patients to the treatment and control groups. This left eighty-nine trials. What followed was months of careful statistical analysis, so that each trial contributed appropriately to the final result. For example, the result of a very small trial would carry very little weight in the overall conclusion, because the reliability of a trial’s result is closely linked to the number of participants in the trial.

The meta-analysis was eventually published in September 1997 in the
Lancet
. It was one of the most controversial medical research papers of the year, because its conclusion endorsed exactly what homeopaths had been saying for two centuries. On average, patients receiving homeopathy were much more likely to show signs of improvement than those patients in the control groups receiving placebo. The paper concluded: ‘The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo.’ In other words, according to the meta-analysis, homeopathy was genuinely effective.

Not surprisingly, Linde’s conclusion was questioned by opponents of homeopathy. Critics argued that his meta-analysis had been too lax, inasmuch as it had included too many trials of relatively poor quality, and they feared that these might have biased the overall conclusion in favour of homeopathy. Homeopaths responded that there had been a quality threshold, which Linde had implemented specifically in order to exclude poor-quality trials. Remember, Linde had included only those trials that were placebo-controlled or randomized. Nevertheless, the critics were still unhappy, as they maintained that the quality threshold had not been set high enough.

Because poorer-quality trials are more likely to produce misleading results, researchers have developed techniques for assessing quality and weeding out those trials that should not be taken seriously. For example, the Oxford quality scoring system, developed in 1996 by Alejandro Jadad and his colleagues at Oxford University, can be used to give a trial a score between 0 points (very poor) and 5 points (rigorous). The system works by awarding or deducting points according to what appears in the published version of the trial. So if the research paper confirms that there was randomization of patients then it receives a point, yet this point can be deducted if the randomization appears to have been inadequate. Or the trial can score a point if the paper describes how the researchers dealt with the data from patients who dropped out from the trial. If the researchers have thought about this in detail and bothered to describe it in their research paper, then it is a good indication of their overall level of rigour.

Critics pointed out that sixty-eight out of the eighty-nine trials in Linde’s meta-analysis scored only 3 or less on the Oxford scale, which meant that three-quarters of the trials were substandard. Moreover, critics pointed out that restricting the meta-analysis to the higher-quality trials (4 or 5 points) drastically reduced the apparent efficacy of homeopathy. In fact, the conclusion of the twenty-one higher-quality trials was that homeopathy offered either a small benefit for patients or no benefit at all. Despite the amount of data available from these twenty-one trials, it was still not possible to distinguish between these two possibilities.

In time, Linde and his colleagues agreed that their critics had a valid point, and in 1999 they published a follow-up paper that re-examined the same data with a special emphasis on the quality of the individual trials. Linde wrote: ‘We conclude that in the study set investigated, there was clear evidence that studies with better methodological quality tended to yield less positive results.’ Then, referring back to the original meta-analysis, he stressed: ‘It seems, therefore, likely that our meta-analysis at least over-estimated the effects of homeopathic treatments.’

Linde’s original 1997 paper had supported homeopathy, yet his revised 1999 paper was much more equivocal. His re-analysis of his own meta-analysis obviously disappointed the alternative medicine community, yet it was also frustrating for the medical establishment. Everyone was dissatisfied because Linde was neither able to claim that homeopathy was effective, nor was he able to dismiss it as a mere placebo.

Despite the lack of clear-cut evidence in either direction, the public was increasingly turning towards homeopathy, either consulting practitioners or buying over-the-counter remedies. This gave researchers a renewed sense of urgency to test the therapy via larger, more rigorously conducted trials. Hence, homeopathy was subjected to a much higher level of scrutiny from the late 1990s onwards.

This eventually prompted Dr Aijing Shang and his colleagues at the University of Berne, Switzerland, to undertake a fresh meta-analysis of all the trials published up to January 2003. The medical research group at Berne, which is led by Professor Mathias Eggers, has a world-wide reputation for excellence and the Swiss government had provided the team with adequate funding for a fully rigorous meta-analysis. Hopes were high that Shang would at last be able to deliver a reliable conclusion. Indeed, after two centuries of bitter dispute between homeopaths and mainstream medics, Shang’s meta-analysis was destined to decide, at last, who was right and who was wrong.

Shang was ruthless in his demand for quality, which meant that his meta-analysis included only those trials with large numbers of participants, decent blinding and proper randomization. In the end, he was left with only eight homeopathy trials. After studying the data from these eight trials – the best available trials on homeopathy – his meta-analysis reached its momentous conclusion. On average, homeopathy was only very marginally more effective than placebo. So, did this tiny marginal average benefit suggest that homeopathy actually healed patients?

Before answering this question, it is important to realize that the results of every scientific analysis are always associated with a level of uncertainty. For example, analysing the age of the Earth gives a result of 4,550 million years, and the error on this age is give or take 30 million years. The uncertainty associated with Shang’s estimate on the efficacy of homeopathy was such that his conclusion was wholly compatible with the judgement that homeopathy acted as nothing more than a placebo. In fact, the most sensible interpretation of the meta-analysis was that homeopathy was indeed nothing more than a placebo.

This interpretation becomes more convincing if we bear in mind another aspect of his research. While conducting his meta-analysis on homeopathy, he also conducted a meta-analysis for a whole variety of new, conventional pharmaceuticals. These pharmaceuticals had been tested on the same illnesses that had been considered for the homeopathy meta-analysis. In this secondary meta-analysis, Shang scrupulously applied exactly the same selection criteria to these conventional drug trials as he had done in his homeopathy meta-analysis. The result of his meta-analysis on conventional drug trials was that on average they worked. Although this result also had an uncertainty associated with it, the average benefit was so large that the effectiveness of these new conventional drugs was not in any doubt.

The contrast between the homeopathic trials and the conventional drug trials was striking. Homeopathy had failed to show a clear benefit for patients and the result was compatible with homeopathy acting as a placebo, whereas conventional drugs had shown a clear benefit for patients, which suggested that they do indeed have a genuine physiological impact on the body. This illustrates the stark difference between pseudo-medicine and real medicine.

Shang published his results in the
Lancet
in August 2005. Based on his meta-analysis, he concluded: ‘This finding is compatible with the notion that the clinical effects of homeopathy are placebo effects.’ Reinforcing this point, the
Lancet
ran an editorial entitled ‘The end of homeopathy’ in which they argued that ‘doctors need to be bold and honest with their patients about homeopathy’s lack of benefit’. This sparked major news stories around the world, angering homeopaths who refused to accept the conclusions of Shang’s meta-analysis and the
Lancet
’s accompanying statement. They attempted to undermine the research by pointing out four key issues, but in fact each of their criticisms can be easily addressed.

1. Homeopaths might argue that Shang’s paper indicates a positive effect for homeopathy, and that his meta-analysis therefore supports homeopathy.
   There is indeed a positive effect for homeopathy, but it is very small and entirely compatible with the treatment being a placebo. Shang’s paper is the most comprehensive analysis of homeopathy during its 200-year history, and such a paucity of positive evidence has to be interpreted as a blow for homeopathy. Crucially, Shang’s analysis confirms the results of about a dozen other meta-analyses and systematic reviews published over the last decade, all of which fail to show that homeopathy offers any benefit beyond placebo.
   
 

2. Homeopaths claim that Shang dredged the data, which means that the meta-analysis was conducted in such a way as to bias the conclusion.
   There are indeed many ways to conduct a meta-analysis. Therefore it is possible to ‘dredge the data’ in different ways until the most positive or negative result emerges, but importantly Shang had stated what his approach would be
   before
   embarking on the meta-analysis, and his approach seemed reasonable and unbiased. In other words, the research was impartial because the goalposts were decided before the data was examined, and the goalposts were of a fair size and were not moved once the research was under way.
   
 

3. Homeopaths point out that the meta-analysis included trials for several illnesses, which makes it too crude to say anything meaningful about homeopathy’s ability to treat individual conditions.
   This over-arching meta-analysis was prompted by the fact that there has been no convincing evidence that homeopathy can treat any individual condition. Whenever researchers have conducted systematic reviews of homeopathy for a particular condition, the results have been consistently disappointing. For headaches and migraine: ‘The trial data available to date do not suggest that homeopathy is effective.’ For muscle soreness, the most tested condition: ‘The published evidence to date does not support the hypothesis that homoeopathic remedies used in these studies are more efficacious than placebo.’ For Arnica in the treatment of conditions associated with tissue trauma (e.g. post operative or post-dental), which is the most widely used homeopathic remedy: ‘The claim that homeopathic Arnica is efficacious beyond placebo is not supported by rigorous clinical trials.’
   
 

4. Homeopaths point out that they offer a highly individualized treatment, which is not suited to large-scale trials in which the homeopathic remedy is standardized.
   Indeed, most trials have not been individualized, but there have been trials in which patients were given detailed consultations and either individualized homeopathic prescriptions or placebo. For example, an individualized trial monitoring ninety-eight patients with chronic headaches over the course of twelve weeks led to the conclusion: ‘There was no significant difference in any parameter between homeopathy and placebo.’ Another trial focused on ninety-six children with asthma and looked at their progress after twelve months of receiving individualized homeopathy or a placebo as an adjunct to their conventional treatment. It concluded: ‘This study provides no evidence that adjunctive homeopathic remedies, as prescribed by experienced homeopathic practitioners, are superior to placebo.’