A Critique of the New America Foundation's Recent NSA Report

I am a big fan of Peter Bergen. His book, Manhunt, about the search for Osama Bin Laden is one of the most useful and informative and gripping reads on a counterterrorism matter I have come across in a long time. So it's with a bit of a heavy heart that I say that his recent New America Foundation report, entitled "Do NSA's Bulk Surveillance Programs Stop Terrorists?" is fatally flawed. Written with David Sterman, Emily Schneider, and Bailey Cahall, the report has grabbed headlines for its arresting conclusion that "the government's claims about the role that NSA 'bulk' surveillance of phone and email communications records has had in keeping the United States safe from terrorism shows that these claims are overblown and even misleading." The report examines the cases of 225 individuals and finds that "Section 215 of the USA PATRIOT Act appears to have played an identifiable role in initiating, at most, 1.8 percent of these cases" and "Section 702 of the FISA Amendments Act played a role in 4.4. percent of the terrorism cases we examined." Other NSA surveillance "played a role in 1.3 percent of the cases we examined." In total, "NSA surveillance of any kind, whether bulk or targeted of U.S. persons or foreigners, played an initiating role in only 7.5 percent of cases." There are a number of problems with the New America Foundation report, mostly relating to the composition of the dataset it examines. That dataset includes 225 individuals but importantly, it includes many fewer events, since a terrorist plot or fundraising scheme will often involves several different individuals. The individuals in the dataset include people "in the United States, as well as U.S. persons abroad, who have been indicted, convicted or killed since the terrorist attacks on September 11, 2001." The dataset "seeks to include all American citizens and residents indicted for crimes who were inspired by or associated with al-Qaeda and its affiliated groups, as well as those citizens and residents who were killed before they could be indicted, but have been widely reported to have worked with or been inspired by al-Qaeda and its affiliated groups." Here are four problems: First, if you're looking for NSA impact, a database of US citizens is the wrong place to look. NSA, after all, is a foreign intelligence agency that isn't allowed to operate domestically, as a general matter. The 702 program specifically bars the targeting of US persons, requiring that targets be reasonably believed to be non-US persons overseas. Yes, the bulk telephony metadata program involves domestic and one-end-in-US acquisition of metadata, but that is specifically to assess whether domestic plots might have a foreign nexus and whether foreign plots may have an agent operating in the US. This is not an agency that investigates US nationals. So looking at a large database of US nationals and finding limited NSA involvement says very little about the effectiveness of the programs in question. Second, NSA is not a law enforcement agency, so looking at a group of criminal cases is also the wrong haystack in which to find the needles at issue here. The perfect NSA operation would be focused on identifying and disrupting some activity overseas and might never result in an indictment in a US court. So looking at a bunch of criminals cases and finding a minimal NSA footprint is a little bit like looking at a bunch of criminal cases and finding that the local fire department has been ineffective because its work only shows up in a few arson cases. Arson is only one small part of firefighting. And the criminal case is only one possible disposition for an intelligence agency---and not necessarily the most prized disposition. This problem is compounded by the fact that NSA material will generally not be admissible evidence (there are exceptions to that rule, but it's a good rule of thumb) and the government generally won't want to release it in the context of criminal cases (there are exceptions to that rule too, but it's also a good rule of thumb), so by the time you have a criminal case developed, the prosecutors will typically be presenting evidence that the FBI generated. This may follow a tip or lead from NSA, but the those tracks---at least when all goes well---will tend to be covered. Third, the New America Foundation report is focused on how a criminal case gets initiated---that is, how authorities first learn of a particular plot. I'm not sure what the right metric is here, but I'm sure that's the wrong one. Information can be critical at many different stages of an investigation---whether a criminal or a national security investigation. If I were, say, the FBI, and a bomb went off at the Boston Marathon, it would be very useful and important to me to run the Tsarnaev brothers' telephone numbers through a database that could offer a window into whether they were having significant overseas contacts. It would be important even after I had identified them as the likely perpetrators, because it could provide insight into whether the bombing was a domestic matter or involved international terrorist groups. It might address the question of whether the AUMF applied---a deeply important in its own right. It might also address the question of whether confederates were still active. None of this has anything to do with how the investigation initiated, but it could be critical important to how it proceeds. Information---both in the form of evidence in criminal cases and in the form of intelligence in non-criminal investigations---is cumulative, not binary. Finally, large numbers of the events in the New America Foundation database predate the authorities whose impact the group is studying. The list of events, available here, includes 105 separate incidents (as opposed to individuals). Of these, 24 predate 2006, when the law first enabled the current iteration of the 215 program. And quite a number more predate 2008, when Congress passed the FAA (including Section 702). This is not in and of itself devastating to the report, since metadata collection was taking place prior to 2006, as was overseas targeting of the communications contents of non-US persons overseas. But it does indicate a certain carelessness about what cases the database includes and excludes. To put the matter simply, the New America report is evaluating intelligence programs mostly directed at non-US persons overseas on the basis of the initiation only of law enforcement cases directed at US citizens, and it is looking at a body of cases that don't correspond temporally to the existence of the programs it seeks to evaluate. As I say, I have a lot of respect for Peter Bergen, but this seems to me a serious misfire.