Avoiding Mandatory Transparency’s Pitfalls in Online Safety Legislation

On April 19, Sen. Dick Durbin (D-Ill.) introduced a new bill, the STOP CSAM Act (an acronym for “child sex abuse material”). It’s the newest in a line of recent bills that attempt to promote greater safety online, in part by demanding more transparency from the providers of popular internet services. These bills all face a common dilemma: how to enhance public understanding of providers’ internal practices and policies without undermining the legislation’s central safety goal.

Durbin’s bill means to impose greater transparency and accountability obligations on major online service providers by requiring them to publicly explain their child-safety policies and practices. Large providers (defined as those with more than 1 million unique visitors monthly and more than $50 million in revenue during the preceding year) would have to file annual reports with the U.S. attorney general and the Federal Trade Commission (FTC) describing how they fight child sexual abuse and exploitation (CSAE) on their services. Those agencies must publish the reports, and may make (or the provider may request) redactions in the published versions. 

According to a summary of the STOP CSAM bill (full text of the bill available here), large providers’ transparency reports must “describe their efforts to promote a culture of safety for children on their platform.” This provision situates the STOP CSAM bill within the present vogue for imposing obligations on big tech companies to provide greater transparency into how they moderate content on their services. Other current proposals include the Kids Online Safety Act (KOSA), Internet PACT Act, and Platform Accountability and Transparency Act bills in the U.S., as well as the U.K.’s Online Safety Bill and the European Union’s new Digital Services Act. The general idea, though the bills vary, is to force large providers to shed some light on their often-mysterious inner workings: how they enforce their terms of service and moderate user content, their aggregate statistics about moderation actions, and so on.

On the surface, transparency obligations for big tech companies seem attractive. Indeed, I’m sympathetic to the idea: I know firsthand from my own empirical research that it can be like pulling teeth to try to get tech companies to share information. That said, their reluctance is not without reason. 

Making a provider publicly explain how its anti-abuse systems and rules work is like handing a bank’s blueprints to a bank robber. Abusive users of all stripes adapt their behaviors continually to keep offending. The more details they know about a provider’s anti-abuse program, the better they can evade and game it. After Twitter voluntarily shared the source code for its recommendation algorithm at the end of March, a former director at the company warned that bad actors might use the code to game Twitter’s system for identifying those who violate its rules. “You can read this and extract what are the rules that govern how decisions are made,” she said. “Now malicious actors may have ways to subvert the protections Twitter has built.” Similarly, if providers such as Twitter publish their content moderation playbooks, “[m]alicious actors can use detailed enforcement guidelines to work around policy restrictions,” according to an industry association for trust and safety professionals. 

Abusers gaming the system is a particularly grave concern in the CSAE context. CSAE offenders tend to be especially savvy users of technology who are sophisticated in evading detection. That’s why Microsoft’s PhotoDNA tool for CSAM detection is licensed only under nondisclosure agreements and its precise operation kept secret: to keep bad actors from figuring out how it works. However, after PhotoDNA’s compiled library was inadvertently leaked by digital forensics vendors, two researchers showed that it is feasible to generate low-quality reconstructions of images that were part of the PhotoDNA database. These “proof of concept” attacks by well-intentioned researchers demonstrate how the publication (whether accidental or mandated) of confidential information about a system’s functionality can enable its subversion by malicious actors.  

This is a major reason why tech companies can be so tight-lipped about their inner workings: Disclosure of anti-abuse measures risks nuking their effectiveness. In drafting recent legislation to shed light on providers’ internal practices, lawmakers appear to have realized the importance of keeping sensitive information out of the public eye. Consequently, their bills’ reporting obligations have variously taken one of several approaches: redact, relax, or retain. 

STOP CSAM’s strategy is redaction. While susceptible to misuse by private entities and the government alike, redaction is an important tool for balancing competing interests: enhancing public understanding of issues of broad public concern while safeguarding sensitive information and enabling oversight by an authority with access to the full, unredacted documentation. The STOP CSAM bill allows a provider to “request that certain information be redacted from the published reports, and DOJ and FTC may redact any information they deem necessary.” This provision, while it could be misused (for example, to keep the public from learning something that would be embarrassing to the provider or the government), would allow providers to shield their anti-abuse strategies while still complying with their reporting obligations.  

Another approach is to relax reporting obligations. The Internet PACT Act bill, sponsored by Sen. Brian Schatz (D-Hawaii), imposes less-demanding requirements for covered providers’ transparency reports than the STOP CSAM bill does. In describing how a provider enforces its acceptable-use policy, the report (published directly by the provider) must give only “a descriptive summary of the kinds of tools, practices, actions, and techniques used … that does not jeopardize the effectiveness of these tools.” Rather than require providers to divulge specifics to the government that could then potentially be redacted from the public-facing report, Schatz’s bill would make providers publish their own reports and would not demand so much detail in the first place. The public and the government would be on a level informational playing field. 

A third approach is to retain reports against public disclosure entirely. California’s new Age-Appropriate Design Code Act (AADC), which became law last fall and goes into effect in July 2024, requires covered providers to generate “data protection impact assessments” (DPIAs) to evaluate how their products, services, or features could potentially harm children. These DPIAs must be provided only upon written request by the state attorney general, who will not publish them: The AADC designates the reports as confidential and exempt from public disclosure. This approach is the most secretive option. It adds nothing to public understanding of a provider’s practices; rather, its presumptive value lies in making providers go through the assessment process internally (and potentially tweak their designs accordingly).  

It must be noted that malicious users are not the only ones who might misuse a transparency mandate. If private companies have to explain their practices to the state, and particularly how they moderate user speech, then, as my colleague Daphne Keller has observed on Lawfare before, that opens the door to political pressure to change those practices to better suit the state’s preferences. (Redaction, as well as California’s “produce upon request” model, might even worsen this problem: If internal data is kept secret from the public but not the government, authorities could pressure providers without their constituents looking over their shoulders.) For that reason (among others), state-level transparency laws have drawn constitutional challenges around the country, from Texas to Florida, from New York to California’s AADC. For members of Congress to introduce similar mandates before those cases have fully shaken out (one way or another) feels a bit premature.

Nevertheless, this Congress appears determined to take action on online child safety. Schatz reintroduced his bill two days after a Senate Judiciary Committee hearing on protecting children online. April 19 saw the introduction of both the Durbin bill and another online child safety bill, the highly controversial EARN IT Act. There’s also a bipartisan Senate bill that would amend federal CSAM reporting law, plus rumors of the imminent reintroduction of KOSA.

If Congress is dead set on proceeding with child safety bills regardless of constitutional concerns about their transparency requirements, they should at least take testimony from major tech companies’ trust and safety teams to learn about how abusive users probe their systems and rules, looking for loopholes and workarounds, testing boundaries. Their input, born of the grueling experience of fighting CSAE and other forms of abuse, can help the government understand which approach will work best to keep transparency reporting requirements from playing into abusers’ hands.

Tech companies have legitimate, serious justifications for maintaining some secrecy around their internal efforts to fight CSAE and other unwanted conduct on their services. Knowledge is power, and in eliciting providers’ knowledge, an imprudent transparency mandate will empower abuse, from malicious users and the government alike. Recent legislative proposals have adopted different approaches to the dilemma of enhancing influential companies’ public accountability while keeping bad-faith actors from gaming the system at users’ expense. Expert voices—not just in trust and safety, but in how governments pressure providers to moderate disfavored content—could help illuminate which path to follow (if any). In their quest to protect Americans online, members of Congress must take care that their solution won’t backfire and harm the very users they wish to help.