Conceptualizing the Mandate for the Bureau of Cyber Statistics

As I noted in an earlier Lawfare post (co-authored with Chas Kissick), the Cyberspace Solarium Commission has recommended the establishment of a Bureau of Cyber Statistics (BCS). In our earlier post, Kissick and I reflected on several questions that relate to the bureau’s organizational structure, in an effort to advance the discussion on the structure and direction that the bureau should take.

This post moves to the second part of the discussion: the substance of the bureau’s mandate. Answering questions about the substance of the bureau’s work will require wide-ranging consultation with the government, the private sector and non-governmental organizations during the coming months. Through that sort of consultation, the outlines of the BCS can be developed.

In an effort to advance those discussions, in this post I identify some of the most salient substantive questions about the BCS. For instance, as Congress contemplates implementing legislation in this session, it will be asked to define the institution: What shall it measure? Why shall it measure certain things? What authorities should it have? And so on. As with the earlier post, the questions identified here are intended to be neither exhaustive nor definitive.

Mission

I’ll start with the broadest question: What is the bureau’s mission? The Cyberspace Solarium Commission envisions that the bureau would be “the government statistical agency that collects, processes, analyzes, and disseminates essential statistical data on cybersecurity, cyber incidents, and the cyber ecosystem to the American public, Congress, other federal agencies, state and local governments, and the private sector.” Taking the commission’s mandate as descriptive, its tasking identifies three types of data (security statistics, incident data and ecosystem health data) that could be of value to five different possible consumers. Given the reality that different consumers have different needs, the BCS might in theory create as many as 15 different categories or groupings of data streams.

Some of those data streams are likely to overlap. But even if that is true, it would be overly ambitious for a new bureau to start with the objective of delivering all of this data immediately. It could also be too broad a mandate for the BCS, even when it reaches a mature level of development. Thus, the first major question that will face Congress is broadly to define what it expects the BCS to do and to prioritize among these various areas of interest. Should the BCS initially focus on the collection of enterprise security data, for example, or on strategic-level network hygiene data? Should it collect data that would be of principal value to the federal government, or focus initially on data of value to the private sector? There are, naturally, many permutations to this inquiry.

How one conceives of the bureau’s mission will influence the more detailed subsidiary questions identified below, but it is critically important to ask these questions at the outset: What do we want from the BCS? Who are the primary customers, and what do they want?

For my own part, I suggest that—whatever the details eventually decided upon—the bureau’s overall mission should be to define and collect metrics that can inform public- and private-sector decision-makers about risk mitigation and resource allocation. In practice, that means metrics that are transparent, scalable, auditable, usable and widely agreed upon. At this high level of generality, the setting of strategic objectives is relatively easy—the devil is always in the details. But in general the BCS should have two broad mission sets:

• To monitor and evaluate the effectiveness of government cybersecurity programs in driving down cyber risk.
• To evaluate the effectiveness of widely held best practices (such as preventive measures and security controls) both within the government and in the private sector.

Who Is the Target Audience?

It is possible to imagine the utility of cybersecurity data to both private or public actors. The development of a transparent and auditable system of metrics is intended to unite decision-makers across various enterprise and use case scenarios.

Given the breadth of potential consumers of data and the variegated nature of their demands, the bureau will initially have to determine whether the first metrics it chooses to develop are of greater value to government consumers or to the private sector.

The argument for prioritizing government-focused metrics is simple: Doing so eliminates many of the external communications and influence questions that are inherent in government-private interactions. The BCS will reside within the government. With the goal of easing the bureau’s implementation into the government, its primary focus should have government consumers in mind.

This initial focus on government activity is likely to pay early dividends. It would not be an “inward focus” simply to measure the effectiveness of government enterprise cybersecurity. Rather, if the bureau were to initially develop government-focused metrics, its consumers could anticipate a more precise and granular understanding of how successful (or unsuccessful) certain programs are. The government would have a better measurement, for example, of the effectiveness of the work that the Cybersecurity and Infrastructure Security Agency (CISA) does to improve cybersecurity of critical infrastructure, of the National Institute of Standards and Technology (NIST) framework’s impact on cyber risk, and of the value of sector risk management agencies’ programming to manage risk in their sectors.

In the long run, the private sector will likely have a greater need for agreed-upon metrics. A vast majority of America’s critical cyber infrastructure is in the hands of the private sector. Private-sector mechanisms for improving cybersecurity (like a liability regime) will depend heavily on good metrics. While some observers may argue that the BCS should focus on the private sector due to greater demand for those statistics, the primary focus on government consumers seems a wiser course. However, the enabling legislation should be explicit in noting that the initial government focus is a stepping stone to metrics for a wider audience, not an end in itself.

What Metrics?

The next, more detailed subsidiary question is precisely what can or should be measured. Congress, in authorizing the BCS, will want to define the scope of its responsibility and its methods.

Much about this is indefinite. There are three broad categories, with some overlap, into which the metrics could fall: metrics that are epidemiological in nature, assessing the hygiene of the cyber ecosystem at a high level of generality; metrics that are more detailed, broadly applicable measures of enterprise security; or metrics that collect incident data. While these categories may appear similar, they are conceptually and practically distinct. Which are the right metrics to develop and in what order?

Several factors will play into that decision. Here are a few considerations:

First, BCS should have the authority to collect and aggregate already-existing public data from whatever sources currently collect them—whether by lawful free access or, if necessary, purchase. Collecting and cataloging preexisting databases into a single integrated source would be an easy and practical first step. For example, various databases across the United States already collect incident data, such as Verizon’s annual report on the topic. The federal government may currently hold data through other existing programs. Even without creating any new reporting requirements, Congress would make great strides by authorizing the BCS to identify generally available cybersecurity data and aggregate and standardize those reports into a single source. Sources of data for the BCS will likely fall into one of five broad categories: publicly available data, survey data, government-held data, purchasable data and shared private data.

A second factor will depend on the nature of data that is not already created and collected. Some data may be readily observable because it exists, even if not collected in a usable form. In other words, various systems might already produce data and analytics, but entities have yet to affirmatively capture this data. Other types of data probably do not exist but might be created through surveys, a common collection vector used by other federal statistical agencies. One benefit of a survey system is that it would probably not pose many novel legal or political issues in implementation.

Other types of data might require new enterprise implementation to create. There will be a spectrum of feasibly collectable data, from data that falls into the category of “already exists, we just have to find it” to “we have to have people change what they do to create and collect it.” Along that spectrum run questions of intrusiveness, ease of implementation and degree of utility.

In creating the bureau, Congress will also need to consider whether or not the BCS can deal only with existing data or whether it has any role in defining new, useful metrics. This is different from a mandate to create new data—it is the antecedent question of whether or not the BCS can define new and potentially useful data requirements that others (such as the insurance industry) might value as part of their own risk mitigation process. Given the expertise that is likely to be developed at the BCS, this also appears to be an applicable area of effort for the bureau.

A third factor to consider—if the BCS is authorized to develop new metrics—is to determine how those metrics should be developed. Procedurally, some form of public consultation would be valuable. Substantively, one significant area where the bureau can add value to the metrics discussion is through a broader and more holistic view of the network environment. The BCS might be tasked with developing models of network connectivity in order to identify dependencies and predict potential impacts. This sort of modeling will assist in prioritizing risk mitigation efforts at both the strategic and the enterprise levels.

Put another way, that which is measurable is often not meaningful. The BCS could serve a helpful function by working to identify more meaningful metrics that are feasible to collect. That effort comes with an implicit problem of course—once identified there will be economic and political incentives to implement them. Some observers may fear that new metrics that are initially standards could evolve into mandates. Hence, one can anticipate gentle resistance to the effort—but it seems as though Congress can and should make a judgment that the benefits of this effort outweigh the costs.

A fourth factor to consider is whether or not the enterprise-level metrics developed can—or should—be used for enterprise comparison purposes. No matter how the metrics are constructed, there will be some institutions or critics who put them to that use. But it would be a far different thing if the bureau were to specifically undertake a rank ordering or comparative assessment of enterprise security in a sector of the economy. Those sorts of comparisons will be fraught with controversy. This type of “scanning and scolding” would likely be ineffective. Congress should give serious consideration to that question before it authorizes such comparisons.

Another important question for congressional consideration concerns preexisting metrics for cyber incidents and technical compliance factors. Many metrics currently in use concern cyber incidents or other technical compliance factors. The bureau will be well suited to develop a broader, more general and more holistic set of metrics about an enterprise’s security posture. Large enterprises already collect, for example, data about how well they manage updates and patches—perhaps some of that systemic data can be generated more widely and aggregated to a higher strategic level.

A final factor is efficaciousness. Whatever other tasks it may have, the BCS should be obliged to test and validate its own efforts to be useful. To accomplish a self-assessment, the BCS will need to identify validation methods currently in use and which are most appropriate, such as the validation guidance provided by the Food and Drug Administration.

Related Data Authority—Anonymity and Privacy

Two related issues should also be identified, regarding not what data is collected but how it might be collected. Private-sector actors will be more willing to participate in data collection efforts to the extent their own enterprise is not specifically identified. Thus, the BCS will need authority to maintain the anonymity of data it collects, as well as the capability to maintain a secure data storage system. Existing models for maintaining the legal confidentiality of confidential business information will provide a good template for this aspect of the bureau’s work. CISA, for example, has several anonymized information-sharing systems that may provide a useful template.

Similarly, any data collection authority will implicate the privacy interests of those enterprises and individuals about whom data is collected. In authorizing the BCS, Congress will need to address the delicate question of which privacy rules apply to the data. It would do well to import here the privacy rules that are currently used in other statistical agencies. Again, these rules may provide a useful template.

Collection and Analysis?

What, if anything, should the BCS do with the data it collects? Many of the statistical agencies of the United States collect data and publish it—nothing more. But others offer analysis of the data. Will analysis of this data be part of the bureau’s mission? In other words, how much (if any) analysis does Congress want BCS personnel to do? Does the BCS exist primarily to purchase/collect, standardize and share data without analysis? Or is there additional value the bureau can provide analytically? And in the latter case, how much? Put another way, is the BCS a “pure” statistical collection agency, or does it also have a remit of providing analysis that creates a new product for use by government and nongovernment consumers?

Analysis will probably not be an initial priority of the bureau—it will need, at first, to develop its definition and collection capabilities. But even if the analytic function trails the collection function in development, the question for Congress is whether or not analysis is within the broad remit of the bureau at all. If authorized to conduct analysis, there would be an opportunity for the bureau to work cooperatively with partners in this regard. The bureau need not have its own analytic capability, but it could sponsor data scientists and researchers to work with its data for mission-relevant analysis. A grant program, for example, to allow a university to study an existing dataset on breaches might prove valuable.

Regarding the question of definitions, the bureau can also have an effective role when it sets standards and creates an agreed-upon framework for discussion. Some of the biggest successes of CISA/Homeland Security in the past years, for example, have been in defining standardized rules for automated threat information sharing.

Congress will need to determine whether and to what degree the bureau can assist in the development of a standardized lexicon for metrics. Lack of definitions about even the most basic ideas, such as the definition of an intrusion, has been an enduring problem. A common vocabulary in reporting cyber incidents would be quite useful.

Authorities

The final puzzle piece lies in the authorities with which the BSC will be created. The Cyberspace Solarium Commission recommends that the BCS be empowered to collect “aggregated, anonymized, minimized data on cyber incidents” from government bodies and companies “that regularly collect cyber incident data as a part of their business.” It would also receive such data from breached companies themselves, upon passage of a national breach notification law (Recommendation 4.7.1).

The commission does not address the precise manner in which this data will be collected. One can, for example, imagine a collection structure that is mandatory, in which information about cyber incidents and breaches must be reported to the BCS, on pain of regulatory or even civil enforcement. One can equally envision a system of voluntary reporting, perhaps with a structured set of incentives to reward those who choose to report, for their civic-mindedness. In addition, one can imagine a system in which the bureau is entitled to purchase publicly available information (perhaps at a reduced cost). Neither a voluntary, incentive system nor a purchase system seems to intrinsically pose a legislative challenge.

Well before legislation is proposed, however, it seems clear that an effort to give the BCS mandatory authority would be controversial. It may be the only truly controversial part of the authorization process. The lack of a mandatory authority may limit the bureau’s effectiveness—perhaps to a significant degree. Congress needs to decide whether mandatory collection (as already exists in some other federal statistical agencies) is critical to the bureau’s success.

I tend to think that mandatory collection is an important tool. But in the end, even a bureau that can only create incentives for reporting (and which took advantage of preexisting reporting requirements in other contexts) would be better than no bureau at all. As Congress moves forward in authorizing the BCS, it should try to avoid politicizing the debate and converting a discussion about the creation of a new statistical bureau into a fight about the American regulatory state.

Conclusion

In the end, two points seem clear. First, the BCS should focus on data and (possibly) analysis, building on existing cybersecurity frameworks. The precise authorities for the bureau’s collection will be a controversial point that requires negotiation. Therefore, the goals for the institution should be descriptive rather than prescriptive—it should develop and publicize data, while leaving to other agencies the decision about what consequences should flow from that data. While a natural focus on the government first seems inevitable, the BCS should ultimately decide its initial focus based on public feedback.

Second, the BSC has two related broad functions—to build and release useful data sets, but also to establish standards of lexicon and metrics. As the bureau matures, it may consider issuing analysis and standards/guidance for modeling. (This strategy would best be seen as a “down-the-road” function dependent on public feedback.)

The goal of establishing the bureau should be simple—to make cybersecurity metrics actionable for making informed decisions in both the public and private sectors. The creation of the BCS will be a necessary first step to that end.