Wednesday, 7 April 2010

Guest Post: Stephen Curry on Evidence and BCA v Singh

In the first guest post on Jack of Kent, Professor Stephen Curry of Imperial College writes about the nature of evidence and the Court of Appeal decision on British Chiropractic Association v Simon Singh.

Science: taking a closer look

About a year ago when I became aware of the libel suit brought by the British Chiropractic Association (BCA) against Dr Simon Singh, the first thing that struck me—as someone who is a working scientist—was the unscientific nature of the action.

Last week's ruling on meaning by the Court of Appeal has confirmed me in that view.

Many words have been expended on the Court of Appeal's judgement. I have particularly appreciated the clear legal exposition of my host, Jack of Kent, Heresy Corner's dissenting take and a perceptive examination of the BCA's mindset by Dave Gorman.

I would like to expend just a few more words to consider paragraph 26 of the judgement, which seems to me to go to the scientific heart of the matter. Let us look at exactly what it says:

26. What "evidence" signifies depends heavily on context. To a literalist, any primary fact – for example, that following chiropractic intervention a patient's condition improved – may be evidence of a secondary fact, here that chiropractic works. To anyone (and not only a scientist) concerned with the establishment of dependable generalisations about cause and effect, such primary information is as worthless as evidence of the secondary fact as its converse would be. The same may equally well be true of data considerably more complex than in the facile example we have given: whether it is or not is what scientific opinion is there to debate. If in the course of the debate the view is expressed that there is not a jot of evidence for one deduction or another, the natural meaning is that there is no worthwhile or reliable evidence for it. That is as much a value judgment as a contrary viewpoint would be.

There are several interesting points here.

First, it is pleasing to see that their Lordships have a sophisticated understanding of the nature and variability of what constitutes evidence, even in science. I can only speculate that they have read either or both of Ernst and Singh's Trick or Treatment and Ben Goldacre's Bad Science, which give detailed expositions on the factors that determine the quality of scientific experiments, with particular reference to clinical trials. Not all experiments are equal. Controls—the deceptive application of a mock treatment—are vital. Blinding of the scientists and the participants to whether any given participant is receiving the mock or the test treatment provides the most unbiased way of determining treatment efficacy. But not all investigations adhere to these standards.

Second, their Lordships draw an interesting distinction between "literalists" who are prepared to make facile causal links between observations and those "concerned with the establishment of dependable generalisations about cause and effect". This latter group includes but is not restricted to professional scientists. But, given the use of that word "dependable", it is difficult not to envisage that this group must include those who are willing to apply the scientific method rigorously. Few people, I imagine, would be prepared to submit to treatments developed by "literalists".

Third, by highlighting this distinction, their Lordships draw attention to an unhappy fact about the scientific profession: not all scientists are equal. Some of them are cleverer than others. Some of them are more critical in their interpretation of experimental data. And some of them produce work that is not very good.

As a result of this variability, we have the necessary cut and thrust of scientific debate—the free exchange of criticism that is so important to the method—in which scientists robustly evaluate this or that piece of evidence. (For anyone interested in the technical details, there is an interesting case study here.) Just because an experiment has been done does not mean that useful evidence has necessarily accrued. The good stuff is kept—for the time being—and used as the basis for further investigation. The bad stuff is discarded and left to lie, lifeless and unread, in the scientific literature.

Crucially, their Lordships contend that it is within the context of this scientific rough and tumble that assessments such as "not a jot of evidence" can readily be taken to mean "no... reliable evidence". These argumentative evaluations of evidence are part of the widely understood terms and conditions of scientific debate. It is a blessed relief that they have been thrown out of court and returned to their proper place: the public domain.

Paragraph 26

There is an interesting follow-on to this judicial dissection. In the very next paragraph of the ruling, the Court of Appeal mention a revealing example of the BCA's approach to scientific evidence:

27. ...Dr Singh's defence includes, in §8(25), a survey of controlled clinical trials on the efficacy of chiropractic in treating infantile colic, none of which, he contends, affords objective support for the BCA's claim. The BCA, in §9(23) of its reply, relies (among other studies) on a 1989 observational study of 316 children, of which it is said:

"This... measured the number of hours each child spent in crying. It showed a reduction in crying time from 5.2 hours each day to 0.65 hours each day at 14 days. This was a very substantial improvement. There was no control group. However, the study constitutes evidence."

Their Lordships are careful to point out that this is not the only study cited by the BCA in support of their claims. But it is telling that this paper [Klougart et al. (1989) J. Manipulative Physiol Ter, Vol. 12, p281-8] is cited at all*.

The Klougart study is considered by the BCA to constitute "evidence". But there is no control and therefore no way to gauge the efficacy of the chiropractic intervention.

This is not evidence that a scientist worth their mettle would consider to be of any real value. This is not evidence that I would tolerate in an argument from one of my undergraduates. This is not evidence that a child at primary school learning about science at key stage 1 of the national curriculum would consider to be derived from a "fair test".

The BCA is a professional organisation that represents the interests of chiropractors who claimed to be able to treat non-spinal childhood ailments by manipulating their spines. Those are bold claims given our modern understanding of human physiology. That the BCA is happy to cite the paper by Klougart and colleagues as worthwhile evidence supporting chiropractic treatments tells me they are far less critical, far less scientific, than the public has a right to expect them to be.


*This study is the second paper listed in the press release put out by the BCA in June 2009 to demonstrate that there is a "significant amount" of evidence to support the claims that chiropractic could treat childhood ailments such as colic.

The totality of this evidence was subjected to intense scrutiny in the blogoshpere and the pages of the British Medical Journal and found to consist of irrelevant or poor quality trials.


No purely anonymous comments will be published; always use a name for ease of reference by other commenters.


gyg3s said...

Daubert the dog that doesn't bark in the courts of England and Wales.

See earlier post.

zemblya said...

An excellent explanation of what real "evidence" actually is, and hence the difference between correlation and causation. Not simply applicable to the sciences of course-this forensic approach to a complex set of facts is just as appropriate in law (as JoK has pointed out) and in particular journalism. Should be required reading for all reporters and columnists (and not just those writing for the red tops either, before us guardian readers get all smug).

Jack of Kent said...

It is nice to have some good writing on the blog at last.

After all, The Times regards it as one of the top 30 science blogs! (I know.)

More seriously, I think the paragraphs are very interesting from a scientific point of view. It almost reads that the Court of Appeal are creating a class exemption for statements which are assessments of evidence.

Cynical Steve said...

The paper cited by the BCA is bordering on gibberish... some babies with colic get better after 14 days. Wow. Who knew! Lets hope this raises the bar for "scientific" evidence in court in future.

One thing that is not clear to me, does Dr Singh still have to fork out the quoted amount of £200k, or does the BCA have to pay his costs?

Stephen Curry said...

Har, har!

This time next year, you'll be in the top 20 I should wager! ;-)

But, to take your serious point: I do hope you're right about the court seeking to establish a general position on this.

Steve Jones said...

I too find it good that judges don't always see themselves as the arbiters of truth.

One thing puzzled me about the original ruling, by Justice Eady on the need to ascribe a formal meaning of the word "bogus". It's hardly as if Simon Singh's article was a legal document and the word "bogus" was being used in a technical sense; it was a newspaper article with the normal common usage variations of such a piece of journalistic text.

There is a jury in a libel trial, and if they are there to do anything, then it is surely to decide whether the complainant has been defamed as perceived by a typical reader. Surely a judge is no better qualified to interpret the common-usage meaning of the word "bogus" than is a jury. It would be different id this was used in a legal document, but it wasn't.

However, I wouldn't want the issue decided on that - I believe a much more rigorous test of whether a libel has been committed is required than something pivoted about the perceived meaning of an ambiguous phrase. The real problem here is such matters can spiral out of control of costs and consequences for things that are really of little tangible impact. If the BCA had not sued, then something close to precisely nothing would have happened in the way of consequences. As it is, the BCA may yet be the Oscar Wilde of the alternative medicine field (albeit without the wit).

In addition, besides the obvious and, thankfully, recognised difference between the nature of scientific and legal evidence I think I'd observe another difference with science. Legal systems tend not to be comfortable with ambiguity. It is the case that there is often a very fine line between disaster and success in cases, and there isn't much of a grey area that is allowed to exist. The winner is condemned or lionised, often on very fine points of difference. I know the Scots have a "not proven" outcome for criminal cases, and that's something over which English lawyers tend to feel queasy. However, in general, it's black and white - one way or another. No allowance for ambiguity in the result. (Open verdicts in Coroner's cases are also much disliked in my experience).

Stephen Curry said...

Steve Jones wrote:

I too find it good that judges don't always see themselves as the arbiters of truth.

I enjoyed your comment but that first statement prompts me to point out that scientists should not see themselves as arbiters of truth either.

There is a fantastic treatment here by Henry Gee (an editor at Nature on the proper business of science.

I wanted to include it in my post but it's a rather philosophical piece and I couldn't quite figure out how to work it in.

amie said...

Prof Curry: I as a non scientist have always taken your requirement that controls are vital as the bedrock of the scientific method and the gold standard of evidence.
Therefore I nearly fell off my seat when I heard the BCA's QC blithely asserting:

"It is our case that the proposition that only RCT (randomised control trials) constitutes evidence is out of line with any accepted medical opinion"

I wondered if I heard right and waited for the transcript to confirm this. What on earth did she mean when she claimed that this out of line with *any* accepted medical opinion?

Stephen Curry said...

Amie, as you might expect, the QC's wording is very careful.

Though not a medical scientist myself, I am aware that the medical science literature is replete with observational or case studies where the doctor reports on observations made during treatment of an individual or a small group of patients. This sort of thing is published so, strictly, it may be counted as evidence.

However, the point is not simply to enumerate the number of published studies but to consider their quality and meaning (which is what I think their lordships were getting at). You are absolutely right that proper controls are de rigeur everywhere else. In my own field (biochemistry/biophysics), if I were to submit a manuscript for publication without decent controls, it would be rejected quite rapidly and justifiably so.

SteveGJ said...

the QC is putting up a straw man there. No scientist that I know of would claim that the only valid evidence is that produced from an RCT. An RCT has a particular place within evaluating drug trials and is the "gold standard" for such things. However, the connection between smoking and lung cancer was not established through an RCT (quite apart from the ethics it's impossible to achieve). There are also studies where direct causal links can be established and an RCT would not be appropriate.

However, that's a very long way from claiming that even if there are other, high quality, not RCT studies that the BCA's ones qualified.

Also the difference between a QC and somebody attempting a scientific stuydy can be seen here. The QC's job is as an advocate, not a seeker of "truth".

@Stephen Curry

The "truth" word is, of course, a tricky one with many meanings according to context. However, as somebody who studied physics (at Imperial, but many, many years ago) and as, perhaps, the natural science discipline which is most closely modelled on the idea of an objective reality (which I think goes back to Plato), I would absolutely agree that the notion is fundamentally flawed. I would point people to the great Jacob Bronowski's series, The Ascent of Man, and particularly the episode on "Knowledge or Certainty".

I science is anything, it is surely a discipline that tries to delineate the limits of human knowledge. I believe the hostility to science among many faiths is not that science claims the only (although it is often misrepresented that way), but because it seeks to hold "other ways of knowing" to the same mechanisms of evidence and reasoning. I'm not one who thinks that if science has recognised limits that this is carte-blanche for any otgher beliefs to "fill in the gaps".

amie said...

Prof Curry and SteveGJ: Thanks for those clarifications.

Stephen Curry said...

@SteveGJ: there are two very curious coincidences in your comment. I too studied physics at Imperial (82-85) — did we overlap? — and just yesterday I received the DVDs that I had ordered of Bronowski's Ascent of Man!

I agree science is more about delineating limits or, as Henry Gee so eloquently put it in the post I linked to in my comment above, quantifying doubt

Steve Jones said...

@Stephen Curry

I'm afraid we seriously underlapped. My time was from 73-76 which was essentially when I decided the limited scope of my talent and work rate indicated a future in academia would not have been a wise move.

Enjoy the Bronowski DVDs, to my mind just about the best documentary series of its type and, dodgy graphics and grainy 16mm film apart, it that stands up extraordinarily well given that it was shown in the year Brian May gave up his PhD studies to become a rock star.

I once watched with astonished disappointment when a fifty best documentary list on TV (selected by industry professionals) failed to include the Ascent of Man. Unfortunately it didn't seem to meet the selectors criteria, one of which seemed to be the inclusion of generous amounts of domestic strife.