Preliminary Observations on the Utility of Measuring Cybersecurity

Cybersecurity is a bit like obscenity. It seems that we know it when we see it, but we have a great deal of difficulty describing it, categorizing it or counting it. Much as with obscenity, there are some obvious answers on which all can agree—having an “internet of things” system with a hard-coded password of “123456” is insecure by any measure—but there is a vast gray area in between the poles where tradeoffs, cost-benefit assessments, and issues of practicality and scalability lurk.

Of late, I have become increasingly interested in this apparent gap. The problem is that the lack of good metrics is (or seems likely to be) debilitating. My hypothesis is that, when governments, commercial actors and private citizens think about new deployments of cybersecurity measures, they either explicitly or implicitly balance the costs to be incurred (whether monetary or nonmonetary, this includes disruptions caused by changes to the enterprise and the resulting, temporary, reductions in efficiency) against the benefits to be derived from the new steps under consideration. And yet there are no universally recognized, generally accepted metrics by which to measure and describe cybersecurity improvements. Much as the difference between erotica and obscenity is the difference between art and prurience, the assessment of cybersecurity remains more art than science.

As a result, decision-makers (whether they be corporate boards, governmental officials or individual users) are left to make choices about cybersecurity implementation based on qualitative measures rather than quantitative ones. They can, and do, understand that a new intrusion detection system, for example, improves the security of an enterprise, but they cannot say with any confidence by how much it does so. Likewise, enterprise leadership can, and do, say that any deployment of a new system (say, an upgrade to an accounting package) will bring with it the risk that unknown or previously nonexistent vulnerabilities might manifest themselves. Yet, again, leadership cannot with confidence ask to what degree this is so and measure the change in their security posture with fidelity.

Nor, candidly, is it even clear what sort of measures are appropriate in this area. Since much cyber activity is commercial, I begin with the assumption that better metrics are those that can be quantified with some sort of economic valuation (e.g., installing a firewall is worth $X in savings from intrusions). But it may well be that the best (or more readily achievable) metrics are less precisely quantified noneconomic metrics (akin to the comparative ratings provided by Standard & Poor’s, for example, this company’s security ranks B+).

In light of this complexity and uncertainty, it appears to me that the problem of measuring cybersecurity is at the core of sound policy, law and business judgment. It is critical to get right. To that end, the R Street Institute has begun an initiative intended to build a consensus around how to fill that gap. This brief post reports on some preliminary observations that our initiative has noted.

(I hasten to add that these observations are entirely anecdotal—based on various conversations we at R Street have had with actors in the field. While the number of these conversations is sufficiently large that I have confidence that the report offered here is relatively robust, it cannot, and should not, be taken as a quantitative assessment.)

Broadly speaking, there are three rough categories of response to the general idea of improving cybersecurity measurement. They range from something close to complacency to something bordering on despair—reflecting, even at this meta-level, the disarray we perceive.

“We’ve Got This”

The first group of responses to the problem are of the Alfred E. Neumann variety: “What, me worry?” More charitably, there are a significant number of actors in the domain who feel as though they have a good handle on the question of security metrics and don’t see it as a problem of significance. By and large, this group comprises significant top-level actors on the network—backbone service providers, large platform developers, and major hardware and software vendors.

With varying degrees of confidence, many of these actors profess to having good internal measures of cybersecurity that they use to assess their own performance and drive internal investments. To the extent they engage in consumer-facing activities, they, likewise, think that their metrics are adequate to the task.

Significantly, none of them (at least none that we’ve met so far) are willing to share their metrics publicly. This reluctance takes two forms. Some are unwilling to share the actual formulations of their metrics. These companies believe their own “secret sauce” provides them with a competitive advantage in the field. Others, while possibly willing to speak about metrics configurations generally (e.g., “we measure our cybersecurity by looking at the ratio of successful to unsuccessful phishing attacks”) are unwilling to share actual data about their performance. They are, as it were, happy to say, “we measure the number of probes on our system,” but not willing to say, “and last year we had 6,200.”

In both cases, the actors do not see any value in a public reference metric against which to compare their performance, both because they think such a reference metric would be inferior to their own internal metric system and because (not unreasonably) they are concerned that a public reference metric will drive governmental oversight, mandates and, eventually, enforcement.

In short, this group is of the view that they know what the problem is; that they have good, workable solutions to it; and that further work on the issue in the public space will only hinder their efforts.

To this, one can only preliminarily respond that a “trust us” solution is poor public policy and unlikely to be sustainable politically in the long run. It would be odd indeed if infrastructure as critical as the American information and communication technology (ICT) network were left totally beyond assessment. And in the current political environment of a “tech-lash,” the avoidance of any public scrutiny seems an implausible solution.

The Impossible Dream

At the other end of the spectrum lie a group of responses, mostly from academics and some practical cybersecurity professionals, suggesting that the quest for a cybersecurity metric is a fool’s errand. For this group, the structure of the network and the nature of deployments on that network are too dynamic to ever be measured well. They add that insecurity is often the product of the actions of an adaptive and often persistent adversary, not simply a technical vulnerability. Thus, for them, the problem set changes too quickly and in ways that are incapable of prediction, rendering the entire prospect of measurement flawed. Put another way, the view here is that the time scales are incommensurate—the process of developing and revising a security metric takes place too slowly to match the mutation of threats. Threat actors will always, and inevitably, be inside the defenders’ observe-orient-decide-act (OODA) loop and that dynamic is built into the very nature of the domain.

There is some attraction to this perspective. It is certainly consistent with the received wisdom in the cyber domain that attackers will always outperform defenders. And there can be little doubt that the top-down hierarchical nature of many enterprises (most notably the government) means that responses to new forms of cyber threat are far from nimble.

At the same time, however, this view is somewhat inconsistent with the perception (captured well by Richard Clarke and Robert Knake) that significant improvements in security have occurred over the past 10 years and that enterprises today are increasingly cyber resilient and responsive. If it is truly the case that security is better now, then it seems clear that, if only at an implicit level, those who are making this (accurate?) assessment have a metric by which they assert their success.

This assurance may have some limits. Improvements in security posture might only be measured at a macro level in ways that don’t translate to firm-level measurements. It may, for example, be the case that measuring the number of breaches per year is only a good metric across a large enough sample of firms to give statistically significant data. So, if the objective is a metric that works at the enterprise level, the perception of macro improvement may not be an indicator that the objective is achievable. To me, this seems a question ripe for further examination.

More fundamentally, this view—that cybersecurity metrics are impossible to develop—seems, at bottom, to be a claim about the uniqueness of cybersecurity. But why should this claim be accepted? Many other domains involve complex and interdependent risks and vulnerabilities. Think, for example, of the incredible complexity of process safety at a chemical manufacturing plant, where malicious actors can create security risks. To be sure, cyberattacks might depend more on innovation than do malicious attacks against chemical plants. And it may be that measuring cybersecurity requires the measurement of a difficult concept—the rate at which attackers devise new attacks, which essentially requires measuring their rate of creativity.

But it still seems unlikely that cyber actors are unique in this regard. Credit card fraud also relies on a degree of creativity among thieves, yet there are very good metrics for assessing the incidence of such fraud. Over time, sound methods have been developed for assessing fraud (and chemical safety) and reporting improvements (or failures) within the context of a broader policy objective. At least at first blush, we should be reluctant to think that cybersecurity, alone of all human activities, is incapable of being assessed.

Finally, for me at least, this perspective of the impossibility of measurement is just too grim. It is, at bottom, a claim that a fundamental component of the world economy—one that is growing by leaps and bounds and that will inevitably have an ever-larger impact on human progress—is beyond effective measurement and therefore beyond the ability of industry or government to effectively control. Is it really the case that this massive sector of the economy cannot be defined with enough particularity that we can tell whether it is getting more secure or less? On this view, the entire ICT sector is, in effect, a black box, whose safety is ultimately beyond understanding. Perhaps that is so. But there is no reason to accept so fundamental a rejection of reason (and its replacement with little more than technological faith) absent a thorough inquiry into the question.

Great Idea, But …

The final group of responses, naturally, reflects the more moderate middle. It is peopled mostly by end users and operators in enterprises that need to implement cybersecurity measures at the retail level. For them, the prospect of an opportunity to conduct a true cost-benefit analysis and compare solutions across various dimensions would be invaluable. It would allow them to calculate a true rate of return on investment and actually enable thoughtful resource allocation. Cybersecurity metrics would be no panacea, but they would, for this group, be a real value-add to the risk-management toolkit.

For observers with this perspective, the questions regarding metrics are far more practical than theoretical. These observers are concerned with the cost of the effort, for example, and they worry about how to validate the results in a way that is transparent, objective and reproducible. They want to know exactly what will be measured and how.

It is, of course, premature to define what those measures might be. But it is not too early for members of this group to offer some indicative suggestions. For example, one major question that needs examination is whether resilience in response to a successful intrusion is a useful measure—and, more notably, whether it is more useful than a metric that attempts to measure the incidence of intrusions directly. Relatedly, if intrusions are to be measured, that may implicitly (or even explicitly) suggest the need for some definition of what an optimal level of intrusion for any enterprise might be (for surely it isn’t zero, is it?).

Another, perhaps blunter, proxy measurement might simply be dollars spent on cybersecurity—or, perhaps, dollars spent on cybersecurity per number of users, or pegged to some other metric that controls for firm size. Clearly, throwing money at the problem doesn’t guarantee better cybersecurity results, and this measure is less probative than are intrusion or resilience metrics. But some scaling of security based on investment might be a relatively easy thing to quantify. If, as one suspects, cybersecurity spending correlates positively with better cybersecurity results, then this, likewise, might be an avenue to pursue.

Indeed, this idea suggests a further possible mode of analysis. Perhaps cybersecurity spending data are at one end of the sensitivity/probative spectrum (not very sensitive, but not very probative either), while data about intrusions and resilience are at the other end (very sensitive, also very probative). Even being able to characterize metrics in this way might be useful and give guidance to enterprises with cybersecurity deployment questions.

For this group of respondents, then, the tl;dr of the problem statement is “great idea, but how are you going to do it?” To which the only reasonable answer is: “We don’t know … yet.”

* * * *

If at this point you’re puzzled, you have every right to be. One group of expert practitioners says that the quest for good cybersecurity metrics is a phantasm. Another, equally confidently, asserts that the problem has already been, to a large degree, solved. Meanwhile, most people in the middle don’t know the answer to the question and have only begun to think about how the answer might be defined. Which, of course, is why the question needs to be asked.

If good cybersecurity metrics already exist but aren’t public, that raises one set of public policy questions about transparency and accountability. If good metrics can never be found and cybersecurity is destined to remain a qualitative art, rather than a science, that raises a completely different set of policy questions about dependency and indeterminacy. And if (as I suspect is true) the question of metrics is a classically “super-hard” problem with answers that approximate truth but can never be definitively resolved, then that, in turn, poses yet other policy questions of cost, value and benefit.

That, in the end, is why R Street intends to explore this question further. More to come ….