Regulations Targeting Large Language Models Warrant Strict Scrutiny Under the First Amendment

In Moody v. NetChoice, LLC, a Supreme Court majority went out of its way to hold (at least in the form of “guidance”) that the traditional First Amendment doctrines that protect editorial judgment, developed in the analog world of newsprint and parade banners, apply with full force to the content moderation decisions social media companies make when curating vast quantities of their users’ data. This was too much for Justices Clarence Thomas and Samuel Alito, provoking the latter to grumble that the majority had blithely endorsed the notion that “there is no constitutionally significant difference between what newspaper editors did more than a half-century ago … and what Facebook and YouTube do today.” But the majority’s guidance on how to apply old First Amendment doctrine to new information technologies is likely to matter far beyond social media content moderation, since it comes just as governments in the United States and abroad have sought to regulate artificial intelligence (AI) systems directly for the first time.

Notably, lawmakers in the European Union, and especially in the United States, have targeted the creation and dissemination of so-called AI models to address the broader risks posed by the proliferation of AI systems. These initiatives have made few distinctions between different kinds of AI models. But NetChoice should be a warning to U.S. lawmakers that their focus on AI models raises significant, even urgent, First Amendment questions. When applied to so-called large language models (LLMs), such as those that underlie the most popular AI systems in use today (ChatGPT, Claude, Llama, Gemini, Stable LM, and others), laws in the United States should—and likely will—face strict scrutiny under the First Amendment as regulations of pure speech.

To be clear, for the purposes of this article, I am concerned only with LLMs. Where an AI model is purpose-built with the moves of the game Go or the folding of proteins, etc., I assume for the purposes of this article that the extent of its First Amendment protection is the same as the protection generally afforded scientific data of any kind. LLMs, though, present a special First Amendment case because they are models of expressive activity, or “pure speech.”

This is a contestable point, to be sure. But given how LLMs are made and what they are made of, LLMs have a clear claim to treatment as a form of pure speech themselves and the protections of strict scrutiny under the First Amendment. That does not mean that AI systems are beyond regulation. Quite the contrary. But it does mean that U.S. lawmakers should approach the regulation of AI systems with greater attention to what precisely is being regulated in order to avoid seeing their initiatives dashed against the United States’s stringent free speech protections.

A Note on Terminology

An AI model is one of the four components that make up an AI system under both the European Union’s AI Act and the White House’s AI executive order, which in turn incorporates the relevant statutory definition from the U.S. Code. Those four components are inputs, outputs, inference algorithms (AI algorithms), and the data, or model weights, on which those inferences are based (AI models). Described simply, an AI system is a technology that takes some direction (the inputs) to produce some result (the outputs) by making automatic predictions (the AI algorithms) based on the properties of some preexisting body of information (the AI models). AI models, in turn, are said to be “trained” on that preexisting body of information (the training data), which can be corpora of anything from the moves of the game Go to the folding of proteins. And an AI model that is trained on a corpus of speech is called a large language model (LLM).

The following illustrates the respective roles of these four components in a simplified explanation of the “transformer architecture,” which is the most commonly used AI system design that is currently operational. Algorithms first encode user inputs into numerical values called embeddings, which are derived from a data set of predetermined model weights that reflect the statistical properties of the inputs in a corpus of training data. A different set of algorithms (called the transformer stack) then iteratively performs a series of basic mathematical operations between those embeddings and additional data sets of model weights, which correspond to the statistical properties of the training data as a whole. This, in turn, yields a set of probable outputs that yet another algorithm (in a process called decoding) uses to generate a legible output.

The Current Regulatory Landscape

Precision matters because the different initiatives in Europe, China, Japan, and the United States focus on different aspects of AI systems, though not always in so many terms. Europe and China have made the most comprehensive effort at regulation thus far, albeit in service of different policy goals and with substantially different levels of transparency. The EU AI Act, for its part, is a voluminous package of regulations on various uses of AI systems for illicit purposes, transparency requirements on AI-generated outputs, and restrictions on the use, marketing, and dissemination of AI systems with certain capabilities. It also targetes general-purpose AI models directly, including LLMs, with transparency requirements and dissemination restrictions whenever they pose “systemic risks.”

China’s AI regulations are less meticulous than the EU AI Act. They explicitly prohibit any AI output that “endangers national security and interests, damages national image, incites secession, undermines national unity and social stability, promotes terrorism, extremism, promotes ethnic hatred, ethnic discrimination, violence, obscenity, and false and harmful information prohibited by laws and administrative regulations.” And they impose a general mandate that AI systems “[a]dhere to the socialist core values,” which makes them at least as comprehensive in application, though far more dependent on discretion in enforcement. In practice, according to the Financial Times, this requirement has led to strict vetting requirements on AI models and to the publication of an AI system “safety” leaderboard by Shanghai’s Fudan University—which currently ranks ByteDance’s Chinese-language chatbot, Doubao-Pro, as safest.

Japan and the United States have applied a comparatively lighter touch so far, emphasizing best-practices and guidelines to establish norms while at the same time fostering innovation. Japan has avoided direct regulation altogether, instead issuing guidelines that resemble the basic principles of the EU AI Act without direct regulatory oversight. And U.S. lawmakers have also emphasized the creation of various working groups to develop guidelines and best practices over direct regulation.

The main exception on the U.S. side, however, has been lawmakers’ willingness to target the most exotic seeming aspect of AI systems, which are the AI models that underlie those systems. Among the few substantive requirements in the White House’s executive order issued in October 2023 are reporting requirements on the development of AI models, including LLMs, based on the computational cost invested in their development. Last December, the House introduced a bill to empower the Federal Trade Commission to compel transparency in AI model development. And California’s SB 1047 likewise regulates the development and dissemination of AI models that are computationally expensive to produce and/or could have “hazardous capabilities.”

The First Amendment Protection of AI Systems

The fundamentally different nature and the different roles played by the different components of an AI system are meaningful for several practical reasons that line up with existing First Amendment doctrine. And First Amendment scrutiny applied to direct regulations on each will—or at least should—be different.

The First Amendment protection due to the inputs to and outputs from an AI system, for example, raise fascinating questions that are still mostly theoretical. But when it comes to the components that make AI systems unique, AI algorithms and AI models, those arguing for and against the First Amendment’s protection of AI systems have tended to lump the two together and argue their positions from within terms of the existing and unsettled debate over the constitutional protection of computer code writ large. But the generic term “code” obscures important technical differences that can be significant under the First Amendment.

In computer science generally, the distinction between the instructions or “logic” that a computer executes (in other words, algorithms) and the encoded data on which a computer executes its algorithms is fundamental. (A superb and accessible description of the centrality of this distinction in modern computer science and cybersecurity can be found in Scott Shapiro’s “Fancy Bear Goes Phishing.”) The key difference between the two foundational processor designs—the Von Neumann and Harvard architectures—is in their respective methods for distinguishing algorithms from data. And the distinction between algorithms and data (i.e., AI models) is also fundamental to the architecture of AI systems.

The distinction can appear subtle because most, though not all, AI systems in common use today employ the transformer architecture, which, as noted in the simplified description above, abstracts the data on which the LLM was trained into several “layers,” or discrete sets of weights, that require the use of algorithms to have any individual or collective value. But the distinction is nevertheless fundamental to how most AI systems function.

When using the transformer architecture, several of the operations that make the AI system “intelligent,” such as the decoding stage (where the output is generated), are almost entirely algorithmic. Many AI algorithms, to include those offered by OpenAI and Meta, for example, even include a “temperature” parameter that can be adjusted to increase or decrease the unpredictability of the AI system’s outputs by algorithmically injecting pseudo-randomness at the decoding stage. All the while, the AI model usually remains static. And those key differences between code that dynamically produces a result and code that statically represents information affects the First Amendment doctrines that AI algorithms and AI models, respectively, implicate.

To the extent algorithms warrant any protection under the First Amendment, it is because of the First Amendment protection generally afforded to instructions. Though the Supreme Court has never squarely embraced the principle, courts have treated regulations on the dissemination of instructions as a species of “expressive conduct” regulations for at least a half-century. This has led courts to subject government regulations on the dissemination of instructions to the more permissive intermediate scrutiny of the O’Brien test, which looks principally at whether the abridgement of expression is both incidental and no greater than necessary to further a substantial government interest unrelated to the regulation of expression. Applying the O’Brien test, the circuit courts of appeals have afforded regulators considerably more leeway to punish dissemination of instructions on, for example, tax evasion, the manufacture of illegal drugs, or murder-for-hire. And more recently, courts have treated regulations that target the dissemination of certain kinds of computer algorithms—for example, algorithms that circumvent copyright protection—as expressive conduct regulations.

Some algorithms, of course, might constitute expressive activity. In NetChoice, both the majority and Justice Amy Coney Barrett reasoned that expressive activity, to include at least some kinds of content moderation, does not lose its protection by virtue of being implemented at scale through algorithms. And it is easy to conceive of AI systems whose algorithms might readily qualify as expressive activity, such as an AI system that generated a never-ending episode of Seinfeld. But, in the main, where algorithms express nothing more than how to do something, the government will be able to regulate the distribution of algorithms that mediate the doing of that something.

The LLMs (i.e., the speech-based model weights) on which AI algorithms carry out their instructions, however, pose a different problem under the First Amendment. The LLMs themselves are not algorithms. They are data. Those data encode a representation of a corpus composed of “pure speech,” in the sense that the Supreme Court recently described it, as “images, words, symbols, and other modes of expression” used “to communicate ideas.” And direct regulations on the creation, compilation, and dissemination of pure speech are traditionally subject to strict scrutiny, meaning they must be narrowly tailored to achieve a compelling governmental interest.

Consider too how LLMs are created. While the “learning” is done by machine, in the sense that algorithms encode the training data into a model statistically, humans direct what is learned and how. The compilation of a corpus of training data is generally portrayed in the media (and often by the AI companies themselves) as a mindless robot eating the internet. But more thought goes into it than that. Countless academic papers and chapters of doctoral theses (including my own) have focused on the often-subtle judgments that go into curating the data on which an LLM is trained, so that the LLM represents the world in a desired way.

In the analogous copyright context, the law treats compilations of speech (even of speech that is not itself copyrightable) as warranting distinct protection because they reflect the compilers’ “choices as to selection and arrangement.” For a half-century, the Supreme Court has similarly recognized that regulations on the “exercise of editorial control and judgment” are subject to strict scrutiny, or at the very least “close judicial scrutiny” under the First Amendment. And courts are likely to—and should—protect the curation of both what goes into an LLM and how those training data are weighted as expressive activity of this kind.

LLMs, therefore, present a straightforward case for protection as pure speech. If the inputs to an LLM are all expressive activity, and if the editorial judgment that goes into curating those inputs is expressive activity, LLMs should, in short, be no less deserving of First Amendment protection than the sum of their parts.

The Debate Over Large Language Models’ Place Within the First Amendment

To be sure, there are persuasive arguments against treating LLMs as a kind of pure speech. Peter Salib, Doni Bloomfield, and Alan Rozenshtein made a compelling argument against precisely that in the pages of Lawfare. Their argument focused on two aspects of LLMs that they contended took them out of the realm of expressive activity: (a) the automated and non-expressive nature of their creation and (b) their unintelligibility once created. They write:

Model weights will almost never be the expression of any human being. No human writes them: They are created through vast machine learning operations. No human can even read them: It took a group of the world’s leading experts of AI interpretability months, and large computational resources, to even begin unpacking the inner workings of a current medium-sized model. Thus, humans have virtually no meaningful influence over or understanding of what model weights might communicate in their raw form.

This piece built on an earlier piece by Rozenshtein arguing against the First Amendment protection of AI models, including LLMs, because unintelligible data fail to “express ideas,” whereas algorithms at least do that much.

It is not clear, however, under existing First Amendment doctrine, why either the technical mode of their creation or their unintelligibility in raw form prevents LLMs from being treated as pure speech. In NetChoice, the Supreme Court declined to decide whether regulations on the commercial use of algorithmic editorial tools, such as those that social media companies use to moderate content, warrant strict scrutiny. But the Court was explicit that “expressive activity includes presenting a curated compilation of speech originally created by others” even when that curation is done with algorithms to produce massive compilations of other people’s speech. And assuming that remains the law, it is unclear what principled distinction compels a different rule when that editorial judgment is exercised to curate an LLM, as opposed to any other compendium of pure speech. Said differently, why—or at what point—does the process of encoding pure speech into an LLM strip it of its First Amendment protection?

In response, Salib, Bloomfield, and Rozenshtein would likely turn to the second principle they identify as taking LLMs out of the pure speech category: unintelligibility. It is an intuitive point. LLMs do not seem like speech in the constitutional sense because, as Rozenshtein argued in his earlier piece, they “are simply impossible to parse or understand without either running them or, at the very least, using sophisticated computational analysis.” In their raw form, they are just numbers—no amount of professional training could enable someone to discern a representation of the world or a discrete idea by reading those numbers alone. And generative AI systems can use LLMs to express an arbitrary multitude of contradictory ideas.

The difficulty with an intelligibility requirement, however, is fitting it into existing First Amendment doctrine. Nearly all electronic speech is encoded into raw forms—whether it is a CD-ROM’s etches, fiber-optic pulses, or electromagnetic waves—that algorithmically abstract and encode data that must be algorithmically decoded to be useful. LLMs, of course, abstract data to a far greater degree than modes of compression whose purpose is to reproduce a faithful copy of the original data, and LLMs generally do not convey discrete, meaningful ideas in the way that this sentence does. But it is unclear why either fact matters, at least under existing First Amendment doctrine.

That could change, of course. Justice Samuel Alito’s concurrence in NetChoice suggested that strict scrutiny should be reserved for regulations that target the individual expression of discrete ideas. But that would require revisiting decades of settled law, if only because the Supreme Court has thus far not limited the rigors of strict scrutiny to the regulation of expressive activity that conveys discrete, meaningful, or even intelligible ideas.

Joseph Blocher has done very interesting work on the First Amendment protection of nonsense, including music, which does not have a “meaning” in any literal sense. But music is at least as powerful as language in shaping our understanding of the world, to the point that Plato, famously, proposed prohibiting the young from listening to music based on the mixolydian mode (which not coincidentally underlies a lot of hard rock music). And as a kind of expressive activity, the Supreme Court has held that regulations of a song’s music are as subject to strict scrutiny as regulations of its lyrics.

It is also wrong to say that LLMs fail to encode an “idea,” at least so long as that concept includes capturing distinct representations of the world and the ideas within it. That is, after all, what images do. Digital images represent the world with algorithmic encodings that are unintelligible without decoding algorithms. And like an image, an LLM’s representation of the world encodes a point of view that reflects the editorial judgments of its creators. If I train an LLM solely on Fox News transcripts, for example, an AI system that uses that LLM will yield starkly different results than if it uses one trained on MSNBC.

This is why a principal goal of China’s regulatory approach has been to ensure that AI models express the right ideas. According to the Financial Times, the Chinese government achieves this through audits that include reviews of both the AI model’s training data and the kinds of responses it yields when plugged into AI systems. Knowing this, the creators of models in China shape the ideas that their models contain by “weeding out problematic information from training data and building a database of sensitive keywords,” such as “Winnie the Pooh.” In fact, the Chinese Communist Party has released a chatbot with its own AI model that it created to represent the national political philosophy, “Xi Jinping Thought on Socialism with Chinese Characteristics for a New Era.”

To be sure, some AI systems inject these points of view or biases with algorithms. In the case of Google’s Gemini, it appears that Google was reengineering user prompts behind the scenes to add diversity parameters that, in turn, led to risible results. But that was an algorithmic fix for the perceived biases of the underlying models, which were a function of the biases of the training data that were selected by the biased engineers who curated the models.

***

The model weights that make up an LLM encode the discrete, meaningful, and intelligible representation of the world that is contained, often latently, in the pure speech that makes up its training data. Indeed, that is the only reason they are of any use. Just as laws that regulate the color spectrum or resolution of an image, or the modality and tone of a piece of music, would have to survive strict scrutiny, laws that target the size, composition, design, and size of LLMs necessarily entail the imposition of regulatory burdens on the curation, organization, and dissemination of a kind of pure speech. And under existing First Amendment doctrine, such laws are subject to strict scrutiny.

To say that the direct regulation of LLMs warrants strict scrutiny under existing First Amendment doctrine is not to say that LLMs are beyond regulation. Strict scrutiny of laws that directly regulate expressive activity in the national security context has not always been fatal in fact. And there is no principled reason that laws regulating intellectual property, trade secrets, libel, obscenity, and myriad other forms of regulable speech would cease to apply to both the production and dissemination of LLMs.

Treating LLMs as a species of pure speech would also not stymie more targeted efforts to regulate the hazardous uses and capabilities of AI systems that rely on LLMs. As noted above, there are principled and well-supported reasons to treat algorithms as a category of expressive conduct subject to fewer First Amendment constraints on government regulation. It is conceivable that lawmakers could regulate the distribution of AI algorithms, for example, that are designed to behave unpredictably without encroaching on the First Amendment. And laws that prohibit AI systems from generating instructions on the manufacture of illegal drugs, tax evasion, or murder-for-hire would have doctrinal support. But under existing First Amendment doctrine, laws targeting the size, development, and dissemination of LLMs should have to survive strict scrutiny because the expressive judgment that goes into their creation and the expressive activity they encode make them a species of pure speech.