European Security Officials Double Down on Automated Moderation and Client-Side Scanning

It has been a particularly busy month for tech policy in Brussels. While many platform regulation and content moderation observers have been keenly awaiting the results of the trilogue negotiations on the Digital Services Act, the European Commission announced on May 11 a new suite of proposed policies on online child protection. These included a policy paper on “the new European strategy for a better internet for kids,” as well as a draft regulation titled “laying down rules to prevent and combat child sexual abuse.” 

This proposed regulation, which has been dubbed “chat control” by European digital rights groups, has major implications for the state of global content moderation. While governments and interest groups have for at least a decade exerted pressure against companies operating user-generated content platforms (mainly social networks, but also search engines and cloud storage providers) to deploy technical interventions to combat the spread of certain forms of content deemed to be socially harmful (especially child abuse imagery and violent extremist propaganda), this pressure has largely been informal and industry collaboration voluntary. 

For example, many platforms use cryptographic hash-matching tools (such as Microsoft’s PhotoDNA) to “fingerprint” user uploads and match them to lists of hashes collected by organizations like the National Center for Missing and Exploited Children (NCMEC). Hash databases rely on trusted human moderators to upload the initial pieces of content to a database but then work automatically, with virtually every type of content posted on major social networks fingerprinted and matched to these databases at the point of upload. Similarly, following pressure from EU security officials, Facebook, Google, Twitter, and Microsoft set up the Global Internet Forum to Counter Terrorism (GIFCT), an industry organization that developed a hash-matching database now used in various ways by GIFCT’s 16 member firms, including Amazon, Airbnb, Discord, Tumblr, Zoom, Mailchimp, and Dropbox.

The EU proposal seeks to change the tenor of what has largely been a system of uneasy collaboration—underpinned by government pressure and threats of potential future sanctions, yes, but still voluntary and grounded in industry good-faith nonetheless. Instead, the regulation seeks to compel firms to deploy systems for the automated detection and removal of a broad range of content that might foster child abuse, rather than just incentivizing and encouraging the development of these kinds of systems informally. 

Who, What, Why?

The draft regulation comes from a different part of the commission than the rest of the EU’s digital services regulatory efforts. Those have been spearheaded by the Directorate-General for Communications Networks, Content and Technology (otherwise known as Connect). Instead, this child sexual abuse material (CSAM) regulation was developed by the Directorate-General for Migration and Home Affairs (otherwise known as Home). In a marked contrast to Connect, Home has a reputation among policy observers for pushing for national security and law enforcement goals at the expense of privacy and fundamental rights. For example, Home’s proposal for a European terrorist content regulation, which was adopted in 2021 and is slated to enter into force this month, also initially featured stringent automated filtering requirements. These were ultimately removed from the law that eventually was implemented after the trilogue process, but the effort to include them in the first place nonetheless reflects Home’s ties to security officials and its strong public safety agenda, pursued despite clear issues of technical feasibility and potential downstream harms to free expression. 

The key impetus for the proposed regulation is Home’s argument that the existing public-private and civil society status quo for fighting CSAM is insufficient and “that the EU is currently still failing to protect children from falling victim to child sexual abuse.” But as far as empirics go, the data is thin. The directorate-general has backed up its argument with figures indicating that there has been a continued growth in the number of reports of CSAM online platforms, but these figures reveal more about reporting trends than anything else. It is unclear, for instance, how many of these reports refer to new unique pieces of content, or existing content that is being reposted; it also isn’t clear how changes in company practices contribute to these numbers. Home’s key argument, however, as laid out in the explanatory memorandum to the draft and its accompanying press releases, is that existing voluntary measures do not go far enough or wide enough across the tech sector. 

Mandatory risk assessments form the core of the approach that is outlined in the draft. If it were to go into law (after amendment proposals from the European Parliament, the Council of Ministers, and, then, negotiation between those two institutions and the commission) following the general direction proposed in the current draft, platforms operating in Europe would be required to conduct assessments as to the various risks that their services pose for the proliferation of CSAM and then take “reasonable measures to mitigate the risk.”

There is a tension in the draft between this seemingly moderate position and the broader scope that the proposal also seeks to cover. The impact assessment conducted by Home in the lead-up to this draft evaluated five different proposals that could be taken to combat the spread of CSAM more effectively. The first of these involved an approach similar to the current status quo, perhaps with more focus and better resources: “non-legislative, practical measures to enhance prevention, detection and reporting of online child sexual abuse, and assistance to victims[,] … the implementation and efficiency of voluntary measures by providers to detect and report abuse, and the creation of a European Centre on prevention and assistance to victims in the form of a coordination hub managed by the Commission.” 

Instead, the explanatory memorandum to the draft explains that Home wishes to pursue the most stringent of the five options apparently on the table, arguing that it is essential that the law “impose legal obligations on providers to detect certain types of online child sexual abuse on their services,” and to do so across three broad content categories: “known” content that has already been confirmed to be CSAM by companies or child safety groups, “new” CSAM content that has not been fingerprinted yet but should be proactively detected, and “grooming” content (seemingly, messages and other communications that might, in the future, lead to the creation of CSAM). 

The proposal is light on technical details and the question of how this would actually be achieved. The institution orchestrating it all would be a new EU Centre on Child Sexual Abuse, which would work closely with Europol and platforms to implement the regulation and oversee the deployment of new technical infrastructures for CSAM moderation. What kinds of infrastructures would be deployed, however, is left vague: The words “hash,” “hashing,” “hash-matching,” “AI,” “artificial intelligence,” and “machine learning” do not appear in the text. (“Scanning” and “automated detection” are both mentioned superficially in the explanatory memorandum). Instead, the word of choice is “indicators,” which appears more than 70 times. 

The draft states that this new EU center would become the central repository for all of these various “CSAM indicators.” The directorate-general is interested not just in systems that seek to detect CSAM content (which can be hashed and matched) but also in systems that seek to predict child abuse material that hasn’t been discovered by government or civil society yet (as well as systems that try and predict whether text and other forms of communication might be leading to “grooming”). As a result, the draft explicitly outlines a vision in which an EU agency would thus become the main home not only for CSAM hash-lists (in effect providing a state-controlled EU alternative to databases like those run by the NCMEC or the Internet Watch Foundation in the U.K.) but also to other “indicators,” presumably training datasets of CSAM and “grooming” message chains that could then be used to train new classifiers. 

While client-side scanning is not mentioned directly, it is clear that this kind of regulatory approach would provide significant leverage for Europol and other security-focused actors in the EU to argue for platforms to move their automated moderation down the stack, either into apps or into operating systems. User-to-user messaging apps, alongside cloud storage providers, are currently doing the least moderation and seem to be the next frontier for a showdown with online child protection groups and law enforcement. In the most narrow implementation of this kind of regulation, if it were considerably softened after the trilogue process—for example, eliminating the provisions about grooming, which seem fundamentally incompatible with EU intermediary liability law’s continuing prohibition on “general monitoring”—major messengers, including end-to-end encrypted ones like WhatsApp, could be expected to at the least start deploying PhotoDNA within their apps on user messages (they already use the tool to check nonencrypted content, like user profile photos). Perhaps every messenger in the EU would also be required by the regulation to use PhotoDNA or other similar systems. 

Security experts and cryptographers have already outlined their many concerns with this kind of development—and they are manifold, ranging from the violation of principles of security engineering, the various possibilities for new adversarial attacks, as well as normative arguments about the trend creating a slippery slope for future, wider-ranging forms of on-device surveillance. That said, following Paul Rosenzweig’s thoughtful framework for evaluating the trade-offs posed by client-side scanning implementations, some very narrow and carefully deployed hash-matching systems at the app level might be deemed to pose benefits that outweigh the risks. 

Child safety advocates will argue that, given the almost indescribable horror experienced by potential victims of child abuse, and the recurring trauma that can be the result of the continued circulation of this kind of material, it is worth it—and, indeed, they are pushing actively for governments to go much further than this more restrained scenario. While these are difficult and emotionally charged policy conversations, digital rights advocates have nevertheless been clear that going down this sort of path must be done with extreme caution, given the various unforeseen consequences it could have not only for privacy and security writ-large. The same might be said for content governance and content moderation.

As I and some colleagues argued in a 2020 paper, even if emerging large-scale automated moderation systems are assumed to be able to function in a “perfectly optimized” way—that is, without overblocking, systemic bias, or major security concerns, which is a far-fetched vision indeed—they still pose major political problems that aren’t really being discussed in the policy debates pushing for their wider deployment. These systems, as already widely deployed by firms on their end “in the cloud,” are already rapidly increasing the opacity and complexity of the new techno-social bureaucratic infrastructures for governing digitally mediated expression. Digging these systems down the stack, deeper into our devices, promises to only exacerbate these issues and would raise the stakes significantly.

What Chat Control Tells Us About EU Digital Policymaking

There has been a lot to distract tech policy observers in recent weeks. Those who have found the time to follow the new CSAM proposal, however, have reacted with deep frustration. Ella Jakubowska, a policy adviser at European Digital Rights, an umbrella organization of European digital rights groups, called it “catastrophic,” a “trainwreck,” and a “law that attacks the foundations of our democracy.”

The cryptographer Matthew Green, who has been active in ongoing debates around end-to-end encrypted apps and wrote various thoughtful analyses of the client-side CSAM scanning proposal that Apple introduced last year, chimed in with a characteristically sharp dissection of the draft regulation’s various issues. After noting the technical issues underpinning what the commission seemed to be asking for, and the huge implications of what the proposal’s “grooming”-related measures posed for encryption, he argued that the roots of the problem go deeper and are actually a symptom of the EU’s lack of technological innovation and domestic technology capacity: With “few native technology providers of its own, [the EU is] unable to even understand what its asking for technologically.” Jakubowska, a close watcher of EU digital policy, added that she was unable to understand how this kind of proposal happened and why other commissioners did not step in to prevent it from happening. 

I’m sympathetic to the continued exasperation of security, privacy, and human rights advocates who continually need to push back against an endless barrage of poorly crafted legislative proposals seeking to undermine encryption the world over. But I think the key dynamic playing out here is a different one than highlighted by Green—it is less a story of ignorance and more a story of capture and the influence of powerful interest groups. 

The distributed structure of the European Commission, and its more than 20 policymaking directorate-generals, creates huge differences in the competencies and governance strategies that various parts of the commission have vis-a-vis digital policy issues. Connect has a highly capable set of policymakers, and the drafts that they produce, such as the most recent Digital Services Act, are generally more restrained, evidence based, and less likely to feature sweeping demands that are impossible to implement technologically. Home, by contrast, is basically the police. 

This institutional makeup also means that certain parts of the European Commission are vulnerable to lobbying from motivated coalitions seeking to securitize some types of policy. If these groups can’t persuade the parts of the commission that focus on digital policy, they can find a more accommodating ear elsewhere, with commissioners who have closer ties to law enforcement and the security establishment, such as Home. A more interesting question for me is how exactly did Swedish Commissioner Ylva Johansson, and her Home staffers, get galvanized into pursuing this type of CSAM regulation in the first place? The topic doesn’t fit clearly within Home’s remit: For instance, the mandate letter that laid out the key outcomes for Johansson at the start of her term doesn’t mention anything about child safety, focusing instead on the classic issues for the directorate-general like migration, borders, and asylum.

Part of the answer involves understanding that this is not just a European debate. Some of the central actors involved in the proposed regulation are not European but, rather, are U.S.-based advocates and firms. As described in comprehensive recent reporting by the German tech-policy outlet Netzpolitik, Thorn, a U.S. 501(c) (3) organization founded by Hollywood star-turned venture capitalist Ashton Kutcher and his former partner Demi Moore, has been a central force lobbying for the legislation. Thorn and Kutcher himself have actively sought Johansson as a potential ally and marketed their commercial product, Safer, which calls itself the “first comprehensive third-party CSAM detection platform,” as a technical solution through which the CSAM regulation could be implemented. These actors know that the commission is an influential global regulator and that it might be persuaded to pursue regulation that probably wouldn’t fly in the U.S. or other jurisdictions due to the First Amendment and other regulatory constraints. For his part, Kutcher tweeted approvingly after the draft was published, arguing that it “will serve as an inflection point in the fight against child sexual abuse material” online. 

Similar dynamics are not uncommon across other areas of EU digital policymaking. In “Of Privacy and Power,” Henry Farrell and Abraham Newman detail the transatlantic interlinkages that characterized debates around EU data protection regulation post-9/11. Their work examines how security-motivated actors in both the U.S. and EU (such as national security officials, law enforcement, and counterterrorism officials) were able to use each other, and the varying affordances of the political systems within which they operated, to eventually drive the adoption of policies more invasive than the supposedly “privacy-focused” EU would (one would assume) develop. While more work is needed to examine the role of child safety groups on digital policy in the EU and beyond, there appear to be major parallels to the story that Farrell and Newman tell: For instance, they explore the important role of various informal transatlantic fora that can bring together like-minded actors motivated to create policy change. A quick examination reveals that these also exist in this context, and are even more developed, with examples like the WeProtect Global Alliance to end child sexual abuse (which not-so-coincidentally recently featured speeches from Johansson).

If, as European Digital Rights has quipped, proposals like the draft CSAM regulation are leading to a third iteration of the “crypto wars”—the new concerted showdown between governments, industry, and the public that will determine the future availability of encrypted communications and devices—this will not be a solely European battle. A powerful transatlantic coalition is emerging and is actively trying to shift the agenda on what, in the realm of content moderation, should be considered technically possible.