A Meta-Study of Drone Strike Casualties

In his widely discussed May 23 speech at the National Defense University, President Obama acknowledged that “much of the criticism about drone strikes---at home and abroad---understandably centers on reports of civilian casualties. There is a wide gap between U.S. assessments of such casualties, and non-governmental reports.” This gap is wide indeed. The range of public estimates of civilian deaths from drone strikes, at the low end, includes the June 2011 statement by then-White House Counterterrorism Advisor John Brennan that there had not been “a single collateral death” in a year as a result of American drones. At the other extreme, the Bureau of Investigative Journalism, a London-based non-profit organization, puts the number of civilian casualties between 84 and 193 in 2010, and between 52 and 146 in 2011---the years that together encapsulate the period in which Brennan said there had been none. Other voices have also weighed in on this debate. For example, the Pakistani government claimed earlier this year during an investigation by UN Special Rapporteur Ben Emmerson, that “at least 400 civilians had been killed as a result of drone strikes, and that a further 200 individuals were regarded as probable non-combatants.” A leaked Pakistan government report prepared for the FATA Secretariat and obtained by the BIJ states that at least 147 civilians (including 94 children) were killed from 2006 to 2009. Human rights and advocacy organizations based in Pakistan and Yemen have put forth varying estimates. Anonymous U.S. government and Pakistani military sources have leaked information and made statements to the press, further complicating the discussion. And Al Qaeda and the Taliban have been known to exploit presumably exaggerated numbers of civilian casualties---and to deny the deaths of military commanders---for their own purposes. Five studies have played perhaps the most substantial role in shaping the public debate on civilian deaths from drone strikes. The New America Foundation (NAF) and the Long War Journal (LWJ), both based out of Washington DC, have created databases that are cited often by the media, policymakers, and academics. The Bureau of Investigative Journalism (BIJ) has also done work in this space---work challenging the low estimates of its American counterparts. The Columbia Law School Human Rights Clinic (CHRC) weighed in on the discussion with a recount of the number of drone strike casualties in Pakistan in 2011, using the data provided by these three organizations. And the International Human Rights and Conflict Resolution Clinic at Stanford Law School and the Global Justice Clinic at the NYU School of Law teamed up to conduct an investigation into several aspects of the U.S. targeted killing program in Pakistan and to provide a detailed narrative about the law and the policy behind it for the interested observer. This latter report does not offer estimates of its own, but it does present a critique of the others. In this lengthy piece, I’m going to describe in detail the first four studies in an attempt to provide some clarity to the continuing debate over civilian casualties from U.S. drone strikes. I examine the different methodologies behind each study, the criteria and definitions each organization uses to count the casualties, and then provide each organization’s most recent casualty counts. Finally, I discuss how to reconcile the discrepancies between each of the estimates---to the extent any reconciliation is possible---and offer some thoughts as to which methodology may be best. Before diving in, a quick word on terminology: One of the biggest difficulties with this subject is that all the organizations use different terms to describe drone strike casualties---and when they do use the same terms, they don’t necessarily mean the same things by them. The term “militant,” common in both press reports and among some of the groups, does not correspond simply to any known legal category. Nor is it adequate to call someone a “civilian” in order to determine whether that person is a lawful target or not; after all, a civilian directly participating in hostilities can be lawfully targeted under the laws of war. Each organization also uses a different approach when it is not certain in which category to put a person who was killed. Some are freer than others in using terms like “unknown” where there is doubt as to how to categorize a casualty, while some are more eager to make assignments in the face of uncertainty. Such subjective choices contribute in significant ways to the differences between each organization’s estimates.

The Different Organizations and their Numbers

New America Foundation (NAF): Under the direction of Peter Bergen, Director of NAF’s National Security Studies Program, a team of researchers at the Washington DC-based think tank compiles data on drone strike casualties in Pakistan from 2004 to the present. In the organization’s words,

This database seeks to. . .provid[e] as much information as possible about the covert U.S. drone program in Pakistan in the absence of any such transparency on the part of the American government. The data was collected from credible news reports and is presented here with the relevant sources.

NAF also collects data on drone and air strike deaths in Yemen from 2002 to the present. To be precise, NAF does not collect data on drone strike deaths directly, but on media reporting about drone strike deaths. It does not do independent fact gathering, nor does it seek to verify the accuracy of the press reports it uses. It simply counts deaths reported in media sources it considers “credible.” Its methodology is, therefore, entirely predictable, replicable, and has the benefit of consistency. It is also entirely derivative and only as good as the sometimes-flawed reporting that lies beneath it. As the organization explains, the sources it uses to build its database include:

[T]he three major international wire services (Associated Press, Reuters, Agence France Presse), the leading Pakistani newspapers (Dawn, Express Times, The News, The Daily Times), leading South Asian and Middle Eastern TV networks (Geo TV and Al Jazeera), and Western media outlets with extensive reporting capabilities in Pakistan (CNN, New York Times, Washington Post, LA Times, BBC, The Guardian, Telegraph).

It separates casualties into three categories: “militant,” “civilian,” and “unknown.” It labels the dead as militants when “two or more news reports label the dead as ‘militants,’ while others call them ‘people’ or some other neutral term.” It categorizes them as civilians when “two or more media outlets explicitly refer to the dead as ‘civilians,’ ‘women,’ or children’.” And it labels the casualties as unknowns when “a majority of reports do not refer to the dead as ‘civilians,’ ‘women,’ or ‘children,’ but one media outlet does” or when “the various media reports are [too] contradictory” to make a judgment. NAF resolves any conflicts it comes across in its counting using the following standard: When “a media report cites ‘some’ ‘civilians/women/children’ but does not specify how many, and no other media sources provide a specific total, [it] report[s] one third of the total victims referenced in that source as ‘civilians’ or ‘unknowns’.” As of this writing, NAF’s casualty counts for Pakistan stand at: 258 to 307 civilians killed, 1,585 to 2,733 militants killed, and 196 to 330 unknown killed. The total number of people killed is 2,039 to 3,370. The rate of the civilian deaths, in other words, ranges between eight and fifteen percent. Long War Journal (LWJ): The LWJ is a project of the Foundation for the Defense of Democracies, also a Washington DC think tank. Its managing editor, Bill Roggio, leads the effort to document drone strike casualties in Pakistan from 2006 to the present. Roggio and his team classify the dead in one of only two categories: “Taliban/Al Qaeda casualties” or “civilian casualties.” LWJ also aggregates data on casualties from airstrikes in Yemen from 2002 to the present, and it classifies the dead using a similar either-or form: “Al Qaeda in the Arabian Peninsula Casualties” or “civilians.” While the other organizations working in this space have substantial sections explaining their methodologies, LWJ does not. It simply says on its website: “The data is obtained from press reports from the Pakistani press (Daily Times, Dawn, Geo News, The News, and other outlets), as well as wire reports (AFP, Reuters, etc.), as well as reporting from The Long War Journal.” Whenever a strike occurs, Roggio publishes a post (such as this one about a recent drone strike in North Waziristan on July 2, 2013), and updates a number of charts on the organization’s website. In a January 2010 article published on LWJ’s website, Roggio and Alexander Mayer wrote: “[I]t is possible to get a rough estimate of civilian casualties by adding up the number of civilians reported killed from the media accounts of each attack.” I could find no other information about how LWJ makes determinations of militant versus civilian casualties, or what it does in the event of conflicting information in media reports. As of this writing, LWJ claims “2,526 leaders and operatives from Taliban, Al Qaeda, and allied extremist groups” and “153 civilians” have been killed in Pakistan since 2006. This yields a civilian death rate of nearly six percent. The Bureau of Investigative Journalism (BIJ): BIJ describes itself as an “independent not-for-profit organization” whose aim is to “pursue and encourage journalism in the public interest.” It is based out of City University in London, and Chris Woods directed the Covert Drone War investigation until recently, which Alice Ross now leads. BIJ documents drone strike casualties in Pakistan from 2004 onwards, and covert action casualties (in other words, victims of drone strikes as well as “airstrikes, missile attacks and ground operations”) in Yemen and in Somalia. Unlike NAF and LWJ, whose focus is on counting militant deaths, BIJ places a focus on identifying non-militant deaths---which it categorizes as “civilians reported killed” with a subcategory of “children reported killed.” According to its methodology:

For each reported US attack, the Bureau seeks to identify the time, location and likely target, and to present as clear a description as possible of what took place during the event. We also seek to identify the numbers of those reportedly killed and injured, and to ascertain when possible whether they were alleged militants, or civilians. Wherever possible, we include other information on casualties, such as name, gender, age, tribal affiliation and other identifying aspects.

BIJ relies on a greater number of international and local media sources than do NAF or LWJ to arrive at its numbers. It also uses a more complex methodology than do the other two groups, because it incorporates---in addition to the press reports---a dynamic array of sources in its counting: information for example, from three field investigations it has conducted in Pakistan, from WikiLeaks, and from publicly available documents such as lawsuits. When BIJ runs into discrepancies in its counting, it resolves them in the following manner:

Where contradictory accounts occur, we strive to speak with particular journalists and sources about their reports to clarify discrepancies. But where these discrepancies remain we include the contrasting accounts in the datasets’ narratives of each strike. . . . Where the reporting is vague but appears to indicate civilian casualties, we will include the line “Possible reported civilian casualties” in that strike’s casualty figures in the Timeline. . . .Where the reporting is more specific, but conflicts with other reports or is from a single source, we use the formula 0-X in our count of civilian deaths, with X referring to the highest reported number of civilian casualties. . . .This ensures that the minimum total number of reported civilian deaths is unchanged but the maximum total incorporates these possible civilian deaths.

As of this writing, BIJ’s number of civilians killed in Pakistan since 2004 runs from 411 to 890 (the number of children killed is 167 to 197). The total killed is between 2,566 and 3,570. This leaves a civilian death rate that ranges greatly---between as low as twelve percent and as high as thirty-five percent. Columbia Law School’s Human Rights Clinic (CHRC): Columbia’s Human Rights Clinic published a study entitled Counting Drone Strike Deaths in October 2012. Its primary author is Chantal Grut, then an LLM candidate, who was supervised by Naureen Shah, then the associate director of the Counterterrorism and Human Rights Project at Columbia. The stated aim of the report is “to thoroughly examine the data and methodology of the three tracking organizations.” The study does not locate or use any sources of its own, but instead uses the data on which NAF, LWJ, and BIJ rely in order to arrive at its own independent recount of casualties for drone strikes in Pakistan in the year 2011. The study seeks to understand the reasons for the discrepancies between the tracking organizations’ estimates, and to explain the limitations of media reports as a metric to count drone strike casualties. In its recount, “where a hyperlink was broken and appeared to be the source of an organization’s upper or lower casualty figure, [CHRC] tried to re-source the article,” and re-categorized every death as a militant or civilian casualty. The Human Rights Clinic bases its classifications on making a “strong,” “medium,” or “weak” identification of each death. The organization defines a strong identification as one in which “the deceased are individually identified by name, and/or where the reported identification of the deceased is corroborated by an independent investigation.” A medium identification is one in which “there are multiple original sources for the identification of the dead. For example, both anonymous officials and a local resident.” And a weak identification is one in which “there is only one source for the identification. For this purpose, [the study] treat[s] multiple anonymous officials as one source, and plural unnamed residents as one source.” When confronted with issues in the data, the organization takes “a lower figure of 0” in “cases where [it] had concerns with the media sources,” and “[i]n cases where there are conflicting reports about the identity of the individuals killed as either militants or civilians, [its] figures reflect both possibilities.” When I asked for further clarification, Grut responded in an email saying:

[I]n cases where the identity of a person killed wasn't clearly reported or was disputed, we counted it as 0-1 civilians killed and 0-1 militants killed, which I believe is the most accurate reflection of the possibilities. So for example for a strike on Jan 7, we counted 0-6 civilians killed and 0-6 militants, but a total killed of 4-6 (rather than 0-12).

The report found that:

First, despite the strong efforts of the tracking organizations, their estimates of civilian casualties are hampered methodologically and practically. Two of the organizations [NAF and LWJ], according to our independent review of the media sources available, significantly and consistently underestimated the potential number of civilians killed in Pakistan during the year 2011. Second, while some of the flaws we identify can be fixed, others are inherent to the process. . . .

The Human Rights Clinic concludes that BIJ’s numbers were the most accurate and closest to its own estimate. It found that the number of alleged militants killed in Pakistan in 2011 was between 330 and 575, the number of alleged civilians killed was between 72 and 155, and the number of total killed ran from 456 to 661. This produces a civilian casualty rate that, like BIJ’s, also ranges hugely---from eleven percent to thirty-four percent. Stanford Law School and NYU Law School: The International Human Rights and Conflict Resolution Clinic at Stanford and the Global Justice Clinic at NYU released a report entitled Living Under Drones in September 2012. The organizations were contacted by Reprieve, a UK-based human rights organization, about conducting this investigation. Like the Columbia Human Rights Clinic study, this one does not seek to collect data on drone strike deaths itself. It does not even seek, as that study does, to do a recount of the other organizations’ data. Instead, it describes in considerable detail a variety of factors that can lead to the conflicting numbers of drone strike casualties. For example, the study explores the limitations of media sources, and explains how news agencies can report conflicting information about drone strikes:

Those who work for major news outlets and wire services tend to spend more time embedded with military and intelligence officials and are thus more likely to report “official” accounts. Those who are not escorted into FATA by the military rely more on locals and stringers. The result is that different journalists with different contacts get different stories, make different decisions about who to trust, and frequently end up publishing conflicting accounts of each strike.

The report also outlines other considerations that may affect the estimates, such as limited first-hand knowledge of the strikes and unreported strikes. Its most significant addition to the discussion, however, comes from the two investigations in Pakistan that underlie it. The research team conducted

[O]ver 130 detailed interviews with victims and witnesses of drone activity, their family members, current and former Pakistani government officials, representatives from five major Pakistani political parties, subject matter experts, lawyers, medical professionals, development and humanitarian workers, members of civil society, academics, and journalists. . . . Investigations included interviews with 69 individuals (‘experiential victims’) who were witnesses to drone strikes or surveillance, victims of strikes, or family members of victims from North Waziristan.

The Stanford-NYU group conclude with the Columbia Human Rights Clinic: the Bureau of Investigative Journalism’s “data currently constitute the most reliable available source,” compared to the numbers of the New America Foundation or the Long War Journal.

Reconciling the Different Estimates

Because the New America Foundation, the Long War Journal, the Bureau of Investigative Journalism, and the Columbia Law School Human Rights Clinic all have slightly different priorities in the way they count the dead, each organization comes to a different conclusion about who---and how many people---are being killed. To illustrate the differences between each organization’s estimates, I pulled the numbers of U.S. drone strike casualties in Pakistan in 2011 from each report. I used NAF’s categories just for ease, and when an organization didn’t have an estimate that corresponded to that category, I simply left it blank. Columbia’s Human Rights Clinic also did this comparison in their study, but the data below are slightly different from those that it reports. The reason, I believe, is that I used the numbers each organization puts forth now for 2011, and some of the groups likely updated their data in the time since Columbia’s report was published---which noted some errors in their data.

Number of Deaths from U.S. Drone Strikes in Pakistan in 2011

The New America Foundation and the Bureau of Investigative Journalism count deaths from drone strikes from 2004 onwards. Unfortunately, the Long War Journal doesn’t have data available for 2004 and 2005, so I just compared NAF and BIJ’s numbers for reasons of consistency.

Number of Deaths from U.S. Drone Strikes in Pakistan from 2004-Present

The three organizations also aggregate data for casualties in Yemen from 2002 to the present, although these numbers are much more difficult to compare. NAF counts “drone and air strike deaths,” while LWJ counts casualties from “airstrikes” (a term it also uses to refer to drone strike deaths in Pakistan, though it is not clear whether LWJ is counting drone and airstrike deaths in this context). And BIJ’s numbers reflect “confirmed” and “possible” deaths from “airstrikes, missile attacks and ground operations” in Yemen, because of the uncertainty and speculation over whether the Yemeni government or the US has carried out many of these attacks. As a result, I don’t think a direct comparison is possible here, but broadly speaking, NAF reports between 557 and 760 militant deaths and a total of 596 and 832 total casualties. LWJ puts AQAP deaths at 349 and civilian deaths at 82. And BIJ’s estimates are between 15 and 52 civilians killed and 239 and 349 total deaths as a result of “confirmed” drone strikes---and between 23 and 48 civilian casualties and 283 and 456 total casualties as a result of “possible” US drone strikes.

At one level, these numbers seem irreconcilable and demonstrate exactly why this debate is so contentious. The New America Foundation estimates that, at the high-end, seventeen percent of all deaths in 2011 in Pakistan were civilian deaths. At the other extreme, the Bureau of Investigative Journalism and Columbia’s Human Rights Clinic put their high-end estimates at 33 and 34 percent, respectively---a number that is twice as high as NAF’s. That number is also nearly four times NAF’s low-end estimate of nine percent. Most Americans would be, I suspect, appalled to know that one-third of the people killed in drone strikes are innocent victims. Conversely, I suspect that most would be pretty proud of keeping civilian casualties below ten percent in this sort of asymmetric conflict.

Discrepancies like this allow those who participate in this debate to see in the data what they want to see. Proponents of the targeted killing program point to the precision and accuracy of drones in waging war, and NAF and LWJ’s low-end estimates of eight and six percent, respectively, corroborate their claims. Those who are opposed to the program on moral and humanitarian grounds can reasonably point to the BIJ and CHRC’s high-end estimates to substantiate their assertions. Neither side is acting unreasonably. However, reconciling these various numbers is possible. The high-end estimates of NAF and LWJ are broadly consistent with the low-end estimates of BIJ and CHRC---so one could take the zone of overlap as representing an evocative common ground. For instance, NAF’s high-end estimate for Pakistan from 2004 to the present is fifteen percent, while BIJ’s low-range estimate for the same period is twelve percent. In another example, in Pakistan in 2011, BIJ and CHRC estimate as low as eight and eleven percent civilian casualties, respectively, while NAF estimates as high as seventeen percent. Dropping the outlier for that period (the Long War Journal), one could think of the civilian casualty rate as likely ranging from eight to seventeen percent. In my view, BIJ and CHRC have adopted the most open and flexible way to count drone strike deaths. Their methodologies are the most comprehensive, and they endeavor to utilize the widest swath of publicly available information when making a judgment. Although some of this information may not be independently verifiable, these databases are undoubtedly more dynamic---which is crucial in a debate as opaque as this one. But ironically, the result of this preferable approach is a far muddier answer: a range so large that it actually tells us very little about whether drone strikes are killing a great many or a small number of innocent people---or somewhere in between. In that sense, the best methodology only serves to demonstrate how little we actually know about the civilian casualties from U.S. drone strikes.