The Bathtub Fallacy and Risks of Terrorism

Bloomberg economics commentator Justin Fox is tired of being told that his chances of getting killed in a terrorist attack are (much) lower than his chances of slipping, falling, and dying in a bathtub. Implication being—suck it up, people, and quit being such irrational babies when it comes to assessing risks from terrorism.

Having also grown weary of hearing some version of this following terrorist attacks or government security responses, I was interested to find out why Fox (who's a very smart guy) thinks such comparisons should be reckoned a bad form of argument. Why does he think comparing fatal bathtub accidents and fatal terrorist incidents is fundamentally inapt, to the point of calling it "The Bathtub Fallacy"? (For those who don't know his work, Justin Fox is a sophisticated business and financial journalist with a keen understanding of the literatures of statistics and risk assessment, and he's also the author of the outstanding revisionist history of the “efficient market hypothesis," The Myth of the Rational Market, 2009.)

What's “The Bathtub Fallacy,” according to Fox? Following a terrorist incident or government counter-measure, he says (quoting a recent Financial Times (paywalled) column by its principal political columnist, Janan Ganesh), statistics are “dug out to show that fewer Westerners perish in terror attacks than in everyday mishaps. Slipping in the bath is a tragicomic favourite. We chuckle, share the data and wait for voters and politicians to see sense.” To this Fox adds:

Sure enough, a couple of days later [after Ganesh’s column], there was Nicholas Kristof in the New York Times, making Ganesh’s point for him: “The bottom line is that most years in the U.S., ladders kill far more Americans than Muslim terrorists do. Same with bathtubs. Ditto for stairs. And lightning.” Now, I love statistics. I cite them and sort them and chart them all the time. I even wrote a whole column last month about various causes of untimely death in the U.S. But I agree with Ganesh that comparing annual deaths due to bathtubs and terrorism is a mistake.

The urtext of ordinary accident/terrorism comparisons might be the 2009 book by political scientist John Mueller, Overblown: How Politicians and the Terrorism Industry Inflate National Security Threats, and Why We Believe Them. (Note: Mueller has been an occasional Lawfare contributor, along with his co-author Mark Stewart—quite graciously, given the distance between his views and what I'd describe as the mean of Lawfare posts on these topics.) Overblown was widely noticed and debated when it appeared in 2009 at the beginning of the first Obama term, and it has remained a book popular among strands of the American left and libertarian right seeking debunking arguments by which to critique the post-9/11 national security apparatus.

Over time versions of the "bathtub" argument have become a bit of a go-to-meme among some journalists and public policy pundits. It pops up with clock-work regularity each time there's a terrorist attack, whether in the U.S., Europe, or elsewhere. But what’s wrong with it? Maybe it’s correct and useful and not a “fallacy” at all; after all, John Mueller is also a smart guy and he doesn't think it's bad reasoning.

This Readings post focuses on the Fox column, however, and the column gives three reasons why Fox thinks the comparison with bathtub slip-and-falls is mistaken and why terrorism is different (it will surprise no one that I broadly agree with them). The three reasons:

First … terrorism is designed to, you know, sow terror. As Ganesh writes, most people can "intuit the difference between domestic misfortune and political violence. The latter is an assault on the system: the rules and institutions that distinguish society from the state of nature. Bathroom deaths could multiply by 50 without a threat to civil order. The incidence of terror could not.”

Second is that ladders, stairs and bathtubs are undeniably useful. Terrorists, not so much ….

Finally, comparing the incidence of terrorism with that of common accidents is an incompetent and irresponsible use of statistics. Household accidents are lots and lots of small, unrelated events. As a result, while individual accidents can’t be predicted, the overall risk is easy to quantify and is pretty stable from year to year.

Note that each of these reasons aims to show that the risks of the bathtub are different from those of terrorism solely on the basis of the real-world consequences of these events—real-world harms—and not as a matter of each category’s “intrinsic" morality (as many would frame the difference). There might also be (and for many, most, perhaps nearly all of us, there is) an “intrinsic” moral difference between these two, but Fox’s three reasons broadly fit within “consequentialist" rather than “deontological" ethics. They are situated within the standard utilitarian framework of economists—the standard public policy framework that seeks to identify, and to the extent possible quantify, real-world harms which, whatever else one might want to say about a category such as terrorism, serve to establish a common denominator for policy upon which even those with differing moral views can agree.

Hence, when Ganesh differentiates between mere “misfortune” and “political violence,” he does not argue that political violence is different (for purposes of comparing risks) because of its wickedness as such—that is, because of its intrinsic morality. What matters for comparative policy purposes are the harms terrorism brings about (including the knock-on and remote ones it threatens to bring about). There are fundamental differences between ordinary accidents and terrorism—apart from any judgments of their comparative morality.

Taken together, Fox’s three reasons constitute a quite sweeping indictment of The Bathtub Fallacy; together they also encompass, in a few short sentences, a wide range of ethical and social claims. (I will say more in a follow-up post about the first reason—the threat to institutions that undergird a society’s common social life.) Fox’s column goes on, however, to focus on the third reason—what he regards as the bad use of statistics in making these comparisons in the first place:

Terrorism is different. There are small incidents, but there are also huge ones in which hundreds or thousands of people die. It’s a fat-tailed distribution, in which outliers are really important. It also isn’t stable: Five or 10 or even 50 years of data isn’t necessarily enough to allow one to predict with confidence what’s going to happen next year. It’s a little like housing prices—the fact that they hadn’t declined on the national level for more than 50 years before 2006 didn’t mean they couldn’t decline. Meanwhile, the widespread belief that they wouldn’t decline made the housing collapse more likely and more costly.

The conclusion that terrorism is different relies importantly on Fox’s characterization as a "fat-tailed distribution" of risks. Fox cites Nassim Nicholas Taleb’s (of Black Swan fame) extensive writing on this, but then moves on to an interview Fox conducted with Carnegie Mellon University professor Baruch Fischhoff in researching this column. Who’s Baruch Fischhoff, you ask? Well, among (many) other things, Fischhoff is a "past president of the Society for Risk Analysis, past member of multiple national and international commissions on the risks of terrorism and other bad stuff, and author of lots of books with ‘risk’ in the title.” Also, Fox adds, he was Daniel Kahneman’s former research assistant at the Hebrew University of Jerusalem in the early 1970s, and thus someone "present at the creation of the school of psychological research that has shown how bad we humans can be at processing probabilities.”

“People who just look at the average are doing the analysis wrong,” Fischhoff tells Fox. Fischhoff does not think, either, that “it’s irrational to fear terrorism more than falling in the bathtub.” Why? It’s different in terms "of the uncertainty and the shape of the distribution, how well we understand it and the possibility of these large-scale events.” Moreover, Fischhoff adds, in another deceptively simple observation, that “people tolerate risks where they see a benefit.”

While that last point might appear to be a mere truism, it sheds light on how risk analysis fits into cost-benefit analysis. In public policy analysis, at least, risk analysis is relevant because it tells us not just about determinate costs and benefits, but about expected or uncertain costs and benefits, which is at the heart of the most important public policy questions of counterterrorism measures. Fischhoff’s observation identifies an important link between risks and benefits. Moreover, it’s an observation that leads directly to Fox’s second reason for why terrorism is different: Bathtubs offer social benefits, whereas terrorists “not so much.” Because "ladders and bathtubs are clearly useful … we take the risks inherent in using them in stride.” By contrast, terrorists and terrorism are not useful—quite the contrary—and so, beyond mere comparisons of the risks, a rational analysis must take into account relative costs and benefits of that event or, more centrally, that activity taken as a whole category.

That's far from the end of the analysis, of course. Even though terrorists and terrorism confer no social benefit (to take the most obvious next analytic step), the policies one might pursue to curtail or counter those risks independently create costs and benefits, and incur risks of their own. The costs of counterterrorism measures aimed at reducing the risks of terrorism might (even where those costs or associated risks are correctly perceived) nonetheless outweigh the social utility gained from curtailing these terrorist threats (even if they are successful as countermeasures).

None of this is news, of course—but from the point of view of understanding what’s wrong with The Bathtub Fallacy, the use of averages for comparisons essentially shuts down these further, yet essential, steps in the cost-benefit analysis. After all, what’s the point of undertaking all that potentially difficult (and costly) assessment of costs and benefits, if you’ve already concluded that a person’s chances of getting struck by lightning are a thousand times greater than getting killed in a terrorist attack? With those odds—if they’re the relevant ones— we might as well pack up the analysis department and go home.

One could go on from there to many additional and more finely-grained considerations, but it’s enough to say that these further analytic steps tend to turn into standard-issue cost-benefit analysis in the context of certain types of risks. (A good place for the student to start in this area is Matthew Adler and Eric Posner’s excellent 2001 (or 2006 alternative) text, Cost-Benefit Analysis, with which I've had good success in classes.)

From the standpoint of where this discussion began, however, Justin Fox's overall point is that the simple comparison of average risks (bathtubs, lightning strikes, etc., versus terrorism) is inadequate at best as a framework for comparative risk analysis. Actually, it's worse than that—it’s "incompetent and irresponsible." It's bad statistical argument notwithstanding how often the meme shows up in public debate over terrorism and security countermeasures.

The potential negative impact of The Bathtub Fallacy is most visible in the public policy debates over the costs and benefits of forward-looking government security countermeasures and policies, and the risks underlying them. While tradeoffs in security and liberties arising in the long-run with respect to a large population will always be hard to assess, using the logic of averages in that assessment will tend to skew the perception of diffuse, long-run security measures heavily toward minimizing their likely benefits. After all, what government security measure, or ecosystem of security measures, could survive scrutiny if it were accepted, and taken as the central comparative fact, that the chances of an individual U.S. person dying from terrorism in the years 1970-2013 was a mere 1 in 4 million? It matters to public policy to be clear on the problems with such arguments.

(I've written on these issues in the past—see this 2007 article of mine on SSRN. After rereading that article today, I think it contains some useful ideas, as well as serious weaknesses. But it does introduce something not raised by this post that maybe bears mentioning: the philosophical question of the comparison of things one might think "categorically" incomparable or incommensurable— differences of kind and not degree. This is a problem that arises in the law of war. Consider the much-debated problem of proportionality in the jus in bello law of targeting, for example, where the comparison is between two things one might regard as being fundamentally incomparable or incommensurable, civilian harm and military necessity. In the context of the above post, my 2007 article suggests that bathtub accidents and terrorist attacks are fundamentally different—incomparable and incommensurable—where bathtub risks are used as the comparative measure against which to consider terrorist attacks or the benefits and costs of government security countermeasures. The article fumbles, however, on how to address the fact that, incomparable or not in some deep sense, public policy (or the law of war) has to make decisions, take actions, and make tradeoffs. Does the logic of decision mean that they must be considered comparable after all? Strongly recommended as an introduction to this philosophical topic is the outstanding 1998 book edited by Ruth Chang, Incommensurability, Incomparability, and Practical Reason, which includes essays by Elizabeth Anderson, Martha Nussbaum, and Cass Sunstein, among others.)