How to Measure the Value of NSA Programs?

The President says that he has ordered a review of NSA surveillance programs, and one of the challenging aspects of such review will be establishing metrics of success and value.  This is a common challenge with regard to national security policies and intelligence, because “success” is often marked by non-events.  I don’t know just how valuable or cost-effective the recently-disclosed NSA meta-data and other surveillance or analysis programs are, but I’m confident that one of the metrics most commonly cited so far by both sides of the debate – terrorist plots foiled due primarily to a particular tool – is a poor one on which to base policy. Once reports of leaked NSA surveillance programs began surfacing a few weeks ago, much discussion focused on government claims that some number – and the numbers vary depending on who is talking and how they define the terms – of terrorist plots had been foiled by them.  Especially in Congress, backers of the NSA wanted to be able to point to specific plots that had been thwarted thanks to a particular tool, and detractors wanted to show that there were few or no cases in which the government could show that a terrorist attack would have occurred but for its use. While “foiled plots” as a measure of value makes for good political talking points, it is a weak analytical tool.  I’d be surprised if this number were high – and not because I’m necessarily skeptical of the programs’ efficacy – and, indeed, if they were high it might suggest other systemic weaknesses in our counterterrorism intelligence efforts.  Here are a few reasons. First, is a high number of foiled plots even good or bad (in terms of measuring a program’s effectiveness)?  Much of our counterterrorism effort, including surveillance, is intended to disrupt terrorist groups’ activities well before they reach the advanced operational planning stages of a specific attack.  Of course, a high number of specific attacks foiled by a particular program would indicate that it’s been quite valuable.  But a low number might reflect a program’s value, too, in helping to disrupt or interdict terrorist groups’ planning, recruiting, funding and other activities in their early stages. Second, focusing on whether any single intelligence program can be credited with stopping a plot misunderstands the nature of intelligence work.  Effective counterterrorism intelligence is usually accomplished by integrating a variety of tools and streams of information.  A major focus of intelligence reform efforts since 9/11 has been improving such integration of collection and analysis tools.  The intelligence community is supposed to be bringing together multiple tools and information streams in support of each other – and sometimes to validate the accuracy of each other.  Again, it would be remarkable if the government could point to many plots that were foiled singlehandedly by the NSA, let alone a particular NSA program, but it might also suggest dangerously inefficient silos within the intelligence community, as well. Serious and rigorous assessment of surveillance programs' value is important, but number of foiled plots shouldn’t be the focus.