New Doubts Regarding the Lancet Iraq Mortality Study

[postscript added, March 16, 2009, 8:30 p.m.]

Since the Iraq war began, an important question for those closely following the conflict has been the number of excess Iraqi casualties resulting from the war and occupation. Various researchers have attempted to estimate this number. Iraq Body Count has kept a running tab of civilian deaths reported in the Western media and, in recent years, by certain Iraqi government sources. The IBC mortality figure, now at around 95,000 civilian deaths, is undoubtedly low due to its reliance on media reports and Iraqi government figures. During times of intense conflict, many deaths likely go unreported in the media, while there have been numerous inconsistencies in and reports of political manipulation of government figures as it may not be in the  government's interest to admit the extent of deaths from the conflict.

An alternate way to estimate conflict-associated mortality is through the conduct of carefully sampled household surveys counting the number of deaths in selected households and using statistical  techniques to extrapolate to the overall population. Much attention has been focused especially, by myself and others, on the Lancet mortality studies of 2004 and 2006. The first of these studies estimated that there had been approximately 100,000 excess deaths from the war by September 2004. The second study estimated that there were around 650,000 excess deaths through summer 2006. They further found that the vast majority of these excess casualties -- around 600,000 -- were from violence, a stark contrast from most other such conflicts studied where large numbers die from poor health and the breakdown of social organization associated with conflict. "Excess casualties" here means the number who died above that number that would have been expected to die had prewar trends continued and the war and occupation not occurred.

We have recently learned that Gilbert Burham, the lead author of second Lancet study, has been sanctioned by Johns Hopkins for deviating from the approved IRB protocol and collecting the names of many survey respondents, a fact that was implicitly denied in numerous public pronouncements. The school does assert that, as far as they can determine, no one was harmed by this ethical lapse. As a result of this sanction, Burnham has been barred by Johns Hopkins from serving as the principal investigator (lead researcher) on studies involving "human subjects" (live people) for five years. He was also ordered to publish a correction in the Lancet, which has now appeared:

"The Methods section of this Article (Oct 21, 2006) stated that 'Participants were assured that no unique identifiers would be gathered.' Upon review, it was determined that a significant number of the surveys contained names of respondents and household inhabitants. This was a lapse in the authors' obligations to protect participants. However, to the authors' knowledge, the completed surveys remained in possession of the research team at all times and there were no known breaches in confidentiality."

This error, and its possible coverup in subsequent public statements means that, in my opinion, we can no longer rely upon the Lancet II mortality estimates. If one major methodological detail was distorted, we simply cannot know whether other aspects of the study were carried out as stated. Until and unless there is far greater detail on these methods, I do not feel that their estimate of 650,000 post-invasion surplus deaths can be trusted.

Burnham had early last month been censured by the American Association for Public Opinion Research for refusing to reveal details of the study methodology. I must say I find this censure highly unusual at best as Burnham is not a member of AAPOR. I have never previously heard of a professional association investigating, much less censuring, a non-member. However, as the Hopkins investigation shows, the non-cooperation may have been to cover up the methodological discrepancy, rather than for more understandable reasons.

I find this episode deeply disturbing. The issue of the magnitude of civilian deaths in Iraq is a profoundly important one. Given the known political sensitivity of the issue, the Lancet researchers should have been especially careful in the controllable aspects of their methodology. They were not. Rather, they gave ammunition to those who would inevitably attack their conclusions for political or ideological reasons. The result is that we are less knowledgeable about this important question than many of us believed as an important data source is no longer reliable.

While I find David Kane's self-satisfied tone to be disturbing, I must admit that he was more right than I had believed regarding the weaknesses in the Lancet II study. As Kane points out, Burnham's public statements were, in spirit if not in legalistic wording, not accurate.

We are left with several other studies estimating Iraqi casualties. The British ORB polling company estimated as of August 2007

"that over 1,000,000 Iraqi citizens have died as a result of the conflict which started in 2003."

While ORB is a reputable polling company, the faith we can place in these results is weakened due to their failure to publish a detailed methodology;  such information is typically included in papers  published in  peer-reviewed journals, which is one reason researchers typically place greater credence on studies published in such journals. When the Lancet II findings were credible, the ORB study appeared to be a replication of the general order of magnitude of casualties found in that study. With the increased doubts about the Lancet II study, the ORB stands as an outlier. I wish the firm would publish a detailed methodology that would allow better evaluation of their findings.

At the low end, a study conducted by the Iraq Ministry of Health and other Iraq government entities in collaboration with the World Health Organization, estimated 151,000 violent between January 2002 and June 2006. While the authors did not estimate the total number of excess deaths -- nonviolent as well as violent -- presumably because these estimates would be less precise, dependent as they would be on estimates of prewar mortality rates, those estimates would be considerably higher by several hundred thousand. Critiques of this study have questioned whether many Iraqi citizens might be reluctant to admit to Iraqi government-associated researchers that a family member was killed by violence. Thus, it is not implausible to assume that this study is an undercount and constitutes a lower bound. As the Ministry of Health study period ended while some of the most severe violence was still occurring, there have likely been many more violent deaths since then.

Thus, the best guess we can make at present is that at least 200,000 people died through violence since the US-led invasion, and that the true figure may be far higher. Moreover, an additional number that could be in the hundreds of thousands may have died from nonviolent causes -- e.g., lack of clean water and healthcare -- associated with the conflict, but this figure is uncertain. No matter what the correct figures turn out to be, it is clear that far too many have died as a result of this war of choice and subsequent occupation which may have deposed a dictator but which also disrupted an entire society.

Postscript (March 16, 2009, 8:30 p.m.):

Several readers have raised the question as to why the lapse committed by Burnham et al. in this study warrants dismissing the entire study. After all, they argue, the lapse of recording names was an ethical lapse, perhaps, but recording extra information should not affect the results. Let me take this opportunity to clarify my reasoning.

The faith one has in the results of any study depends largely on the quality of the research design and on how carefully that design is followed. In the case of a population-based epidemiological survey like the 2006 Lancet study (Lancet II), even minor deviations from the survey design can have large effects on the results. (Survey research depends crucially on every person in the population having an equal chance of being selected.) As one example, if interviewers used discretion -- beyond that mandated by safety considerations -- in selecting households, it could introduce (probably unintentional and unconscious) bias that would make the findings unreliable. For this reason, survey researchers attempt to maintain strict control over the procedures actually used by those collecting data in the field.

We have been assured for years that the design of Lancet II was carefully followed. Now we hear that the specified design was not followed in a crucial way that may have put participants at risk. Furthermore, the Lancet researchers have for years pointed to those very risks as reasons to deny access to raw data and to withhold crucial methodological information when questioned. The fact that the protocol wasn't followed in a central aspect severely reduces the confidence we can have that the study procedures were carefully monitored.

The Baltimore Sun reported:

"Because of the difficulty of carrying out research in Iraq during the war, Burnham and his team partnered with Iraqi doctors at a university in Iraq. Burnham, working out of Jordan, said he made it clear to the doctors that they could collect the first names of children and adults, to help keep the information straight, but that last names could not be collected.

"When the surveys came back to him in Jordan, it appeared that some had last names. Many were in Arabic. Burnham said he asked his Iraqi partners and was told that the names were not complete, which he accepted. But Hopkins, in its investigation, found that the data form used in the surveys was different from what was originally proposed, and included space for names of respondents. Hopkins found that full names were collected."

This description, if true, supports the assumption that Burnham was in no position to carefully monitor the details of data collection for the study. Further, at its most charitable, it indicates severe communication difficulties with the Iraqi staff that may easily have left him unaware of other possible deviations in procedures. If one is not so charitable, one may wonder why Burnham was told a falsehood, that the names were only first names, and thus what else was distorted. In any case, in the absence of this confidence in the study procedures, we cannot maintain confidence in the study's results.

There is yet another troubling aspect of this incident. The lapse that occurred, recording of full names of respondents reporting deaths from violence in a country undergoing civil war after the Johns Hopkins ethics committee and the respondents were told no names or unique identifiers would be collected, is no trifling error. As Johns Hopkins Magazine reported in its February 2007 issue:

"Concern for the safety of interviewers and respondents alike produced two more decisions. First, they would not record identifiers like the names and addresses of people interviewed. Burnham feared retribution if a hostile militia at a checkpoint found a record of households visited by the Iraqi survey teams."