Lawfare Daily: The Policy Implications of the AI Vibe Shift

[Intro]

Mackenzie Arnold : If we’re truly going to see a transformative tech, this might look a lot more like statecraft of, you know, the U.S. government needing to understand what is exactly going on in specific companies and trying to negotiate deals with them.

Kevin Frazier: It’s the Lawfare Podcast. I’m Kevin Frazier, director of the AI Innovation and Law Program at the University of Texas School of Law and a senior editor at Lawfare, joined by Scaling Law’s co-host, Assembly Member Alex Bores, with Nat Purser, director of U.S. policy at AVERI, and Mackenzie Arnold, managing director of U.S. Law and Policy at the Institute of Law and AI.

Nat Purser: Within the IBO and independent auditor universe, we’re thinking a lot about, like, cryptographic verification techniques and other ways that don’t rely heavily on, like, you know, international trust and coordination to verify that other countries are doing what they say they’re doing within the labs.

Kevin Frazier: Today, we’re talking about the AI vibe shift and what recent reporting on AI risks suggests about the future of domestic and international AI policy.

[Main Podcast]

So, the vibes have shifted in the AI policy conversation in a rapid series of weeks, whereas Mackenzie and Nat, you all have been at this game long before it was cool or perhaps viral in the way it is right now.

But I wanna get a sense, Nat, let’s start with you. Given that you’ve had time at Public Knowledge, dabbling in the tech policy world there, moving to Avery, and getting even deeper into the weeds of AI policy, how have you felt this shift in policy focus and salience over the last few weeks?

Nat Purser: I think it’s been pretty chaotic for us within the auditing and evals universe in particular.

But I think for everybody working on frontier AI governance issues, it kind of feels like our time has come, like our moment has come. And also, with that, a lot of extra scrutiny probably of the frontier AI policy community. Some of that I think, totally warranted and quite welcome.

I think that there’s a lot of good, like, pressure testing going on of some of the ideas in this space, certainly of our backgrounds. But I think it’s been fun. I mean, like, just a year or so ago, and it really was a year or so ago, you, Kevin, Mackenzie, and I were, like, having a round table where we’re talking about preemption, and that was really my first foray into AI policy.

So, I’ve really been doing, like, AI policy for, like, a year and a half. I’m easily, like, probably the, the biggest, like, newcomer to AI policy out of everybody who’s here right now. But going from doing stuff like AI literacy in schools and thinking about data centers to doing this kind of work has been I mean, so exciting, and it does feel like we are building the field right now.

But also, you know, it’s crazy. It’s just crazy. The Signal chats are blowing up all the time. People are always talking about new ideas and putting out new papers. I have a million tabs open, but I’ve been enjoying it quite a bit.

Alex Bores: It, you start with preemption, and look we’re right here discussing it again.

As is something- The more things change the more they stay the same.

Nat Purser: I know. I know.

Alex Bores: Yeah. Mackenzie, there was so much news this summer, obviously. But why do you think now is the time that these issues are breaking into the mainstream? And probably, what do you think was the biggest contributor to that, and what does that mean for where we go from here?

Mackenzie Arnold: Yeah. I- in some ways, if you asked me to predict this earlier in the year, what would be the event that organized everyone, I would not have guessed that it would be sort of the second round of things right after, after Jacob Coxon left, right? That it would be an employee leaving that would sort of precipitate this event.

I think that in many ways was sort of built up to because there had been this string of cyber events, right? And people were already quite activated, were already paying a lot of attention. And so maybe this surprised a lot of people, right? And the things that they encountered surprised a lot of people.

I think we’ve seen in the last couple weeks this cycle where it’s been a lot of people encountering issues for the first time, having very confident takes that they think have solved the problem, and why haven’t people thought of this so far? And then quickly realizing, right, in fact, it’s so hard to predict the trajectory of this tech, right?

And that we all are sort of just restrained by, by the amount of uncertainty that we have. And I think now that everyone’s come back to that point, it’s sort of time to get down to business. People are seeing why you want to have robust incident reporting so you can actually figure out what actually happened, right?

People are realizing that, oh, a lot of smart people fervently disagree over what actually happened and what matters, right? And so, you might need more detail than people would have thought in the beginning, right? And you also, people are thinking, oh is government at all prepared to actually process this and make decisions off of it, and what sort of capacity might be needed there?

And so, it’s been really, in some ways gratifying, to see people sort of work through fr- from first steps of what would I actually need to figure this out, and a lot of them are converging o- on a similar set of policy goals.

Kevin Frazier: All right. So, we know that we had this wild string of events where not only did we learn about Hugging Face and OpenAI, and then we learned about that same set of, or a similar set of occurrences leading to a hack of a German wiki that previously no one had heard about.

Anthropic checks its logs and decides, yes, us too. We weren’t able to contain some of our testing. Meta reports the same, so on and so forth. And now we’re left, as you flagged Mackenzie, in this world in which the Overton window shifted faster than a Japanese bullet train. I mean, things just changed so quickly such that now we see policy being announced on Truth Social on a Saturday morning amounting to things like a AI force.

And I recognize that this is a wildly unfair question because all of that we know is from this Truth Social post as of September twenty-second at eight zero nine AM Central Time. But Nat, let’s start with you. What the heck could this AI force be? We know it was analogized to a space force. What would that mean?

And perhaps more broadly, what does this signal, given that the White House is even pursuing this sort of, let’s just call a spade a spade, wild throwing a spaghetti noodle against the wall and seeing if it will stick approach to AI policy?

Nat Purser: So, I actually think that the Truth Social or tweet, whatever the, the post was maybe more revealing than some people are giving it credit for.

I think that when Trump says in there this specific thing about, however, we will also be looking for bad things in AI, and we can do that very easily with our already existing criminal and civil justice system, this is, I mean, there’s a lot of bipartisan agreement these days apparently that we’ve got all the laws we need.

I think that he- is kind of what the administra- that what Trump is alluding to here. So, I’m not sure if this line here is meant to, you know, throw some, some cold water on some of the work that Senator Cruz or Senator Thune are doing right now alongside other Democratic lawmakers to create various types of AI frameworks.

But for me, that phrase, we can do things with our existing criminal and civil justice system, indicates to me that, like, to the extent that an AI force is an important body or has some real power, it might just be a coordinating body that reduces some fragmentation across, like, cyber through NOSTP, Commerce, NIST.

Maybe that’s sort of the, the goal of this, but I can imagine a world in which this happens, and it’s really more performative and for maybe com- communications, like, benefits, and there’s not, like, a real efficiency gain, or there’s not additional clarity created. So, I have, like, low expectations, I think, for what an AI force would look like, but I don’t know if we’re gonna see any, like, revolutionary new law proposals or you know, anything, like, super substantive.

Mackenzie Arnold: I think we’re all kind of reading the same Truth Social leaves here. A- but I do wanna, like, take this at its word, right? What would be the most exciting version of this? And I’m like, the, the version that this admin might see more clearly than the average one is the need for some sort of analysis or intelligence function, right?

It’s an admin that sees things in terms of deals and negotiations, and that’s a, a function that you need in addition to whatever regulatory power that you have, right? If we’re truly going to see a transformative tech, this might look a lot more like statecraft of, you know, the U.S. government needing to understand what is exactly going on in specific companies and trying to negotiate deals with them.

And if, you know, nice branding and some good graphics is what we need in order to get there I’m down for it. I, I think we’ll need this function in one way or another.

Kevin Frazier: Well, so now I gotta know, and this is an open question, and perhaps an open contest for the Lawfare listeners. We know that the Space Force refers to its members as guardians.

What would we refer to as AI Force members, right? Open contest. Whoever wins gets a breakfast taco in Austin.

Nat Purser: Oh, God.

Kevin Frazier: Come down this way and I’ll buy you some breakfast tacos. Nat, I wanna stick with this idea of existing law that you flagged in the Truth Social post because this has become a debate-

Nat Purser: Yeah.

Kevin Frazier: … that’s even more salient. It’s been a debate for a long time. Scott Brennen and I at NYU, apologies for the shameless plug, the two of us worked together on an analysis for Lawfare, I think almost 18 months ago, looking at how existing consumer protection law maps onto some of the AI risks, but you flagged in a excellent tweet that there’s a big difference between the idea that, hey, there’s no exception for AI for existing law, right? If something-

Nat Purser: Right ...

Kevin Frazier: …if fraud’s illegal, it’s illegal if you do it with AI or not. If discrimination is illegal, it’s illegal if you do it with AI or with not. But there’s also an argument to be made that there are some AI risks that aren’t covered by existing law. Can you explain more why that matters in this debate?

Nat Purser: Yeah. I think that, like, if you commit to the position that we don’t need new AI laws, I mean, you’re committing to a position that, to be clear, would preclude you from, like, forming a new AI agency, potentially and creating stronger oversight laws. I think a lot of the people who are saying these talking points are frankly the kinds of people who would like to see some of those things happen in 2029.

So, I think if y- if you go out there and you say something to the effect of like, well, we’ve got all the laws that we need, you’re gonna find yourself in an awkward position in a couple years when we potentially have a change in a- administration. I’m worried about things, like I understand what these people are trying to get at, I understand why these people think that this is in some ways, like, owning the AI executives or the AI safety community.

And I think that they view those groups as interchangeable, which is another important point. But I think at the end of the day, you really don’t wanna commit to that position, and there’s plenty of recent incidents that are not covered by law that we should try to be addressing through novel pieces of legislation.

Kevin Frazier: Yeah, and one thing I’d flag for folks too is, like, this Hugging Face OpenAI incident, for example. A lot of folks were saying, oh, well, we’ve got the Computer Fraud and Abuse Act. Clearly no. Clearly that’s going to be sufficient. And yet, you know, th- this is why diving into the fact that intent is often such a core component of criminal and even in some cases civil liability, it’s really hard-

Nat Purser: Yeah.

Kevin Frazier: … to say that anyone at OpenAI intended for their agents to hack Hugging Face, and so the CFAA and related law just may not address that harm. But I know, Alex, you’ve got some thoughts on the application of state law that we wanted to dive into.

Alex Bores: Well, I think, you know, I have a very biased view on state laws here but Mackenzie, you made a great point on Twitter about companies already have a way to make some of their commitments binding, right?

We have SB53 and now RAI’s coming online, so while we have some people saying there are no federal laws needed, where are we on state laws, where are we on implementation, and what do you expect from states next year?

Mackenzie Arnold: Yeah. So, there’s kind of the good, bad, and the ugly for state laws, right? The good part, and what I’m talking about on X, right, is the RAISE SB53, SB315, a- all of these create a mechanism whereby you can create voluntary agreements.

You can choose what’s in them. You can choose for it to be a mostly blank piece of paper that maybe has some section headers. But if you put it in there, you’re bound to it, right? If you make a promise, it is a promise, you’re gonna be held to it. And one of the things that, you know, companies conveniently omit when they’re making all of these, you know, framework announcements and blog posts and other things is, hey, we’re doing this on our blog.

We have a pla- we, we could make these commitments in the framework. We could make it binding, right? We could put our money where our mouth is.

Nat Purser: Right.

Mackenzie Arnold: That’s not what they’re using right now.

Kevin Frazier: This, the infamous blog loophole of oh- ... we just put it in a blog. You know, you can’t hold me to a blog.

Mackenzie Arnold: E- essay culture, man. Then in, in the middle, right, I think what really surprised people, and what you’ve also seen me talk about here, is people expected that incidents like this would be covered, right? We argued over these laws for years, and it- the terms in which they were talked about was as though they were going to be the most onerous, most difficult things for the companies.

And then you get your first incidents, and it involves the autonomous hacking of agents and you not noticing it for extended periods of time, and lo and behold, they’re not probably covered, right? That, that-

Nat Purser: Yeah.

Mackenzie Arnold: ... that’s really startling. And it has to go back to this, i- in some ways everyone got a little too clever, and the ways that you qualify for reporting under this are these really kind of like quasi-metaphysical, complicated, you know, really bad things have to happen, or you have to show that it was deceptive in nature and the deception w- was aimed at the developer.

We don’t wanna answer all those questions at threshold, right? You’re getting the reporting because you’re trying to figure out if those things actually happened or whether you’re concerned, right? And so, I think we’re all realizing that in fact so for example, with incident reporting, the real answer’s gonna be more complicated than this.

You’re gonna have some tiered system that says at least for initial notification or reporting, the bar should be way lower, right? It should be, this is something that seems generally concerning. We’d like to have more information about it. And then if you have some of those plus factors, right, if people are actually harmed in the real world, if large amounts of money are lost, then maybe what you get is additional investigative powers or something like that.

And that’s not where the, the current state of state law is, right? Most of these things don’t qualify. Even if they do, all you’re getting is a plain language summary of what happened. That’s not gonna satisfy anyone.

Alex Bores: How does OpenAI’s recent disclosure framework fit into sort of those tests that you just laid out?

And I actually don’t know if that was on their blog or if that was official, but, you know, what was your take on they, they put out some pretty specific when they would disclose or not. What was good there and what would you love to see improved?

Mackenzie Arnold: Yeah. So, this wasn’t integrated into their binding framework, right?

So, that’s number one, right? Why not put it in there? The other part is, it’s really focused, th- this framework, on disclosing information that your employees raise to you, right? So, if someone at OpenAI sees something concerning, they raise their hand, it goes up the chain, it’s saying, okay, we’ll have a process, we’ll review it, we’ll reveal it if it has certain criteria to it.

Right? You can quibble over the specific criteria. They’re actually, I think, pretty good as a starting point. The bigger problem in my mind is that wouldn’t have actually solved what happened over the summer, right? What happened over the summer is that it sounds like employees did not escalate this information, right? That it did not make the rounds, right?

And if all you’re doing is sort of reacting to what information comes up to you, a lot of these things are gonna go undisclosed. So, a- along with any sort of disclosure framework, I think you also have to be thinking about monitoring, including automated large scale monitoring that can actually keep track of what are, you know, a large number of agents being trained or evaluated at any given time, where you can’t, you know, have human visibility into each and every thing.

So, that would be one of my main improvements to this, is make sure you actually get the information so that you’re disclosing it.

Kevin Frazier: And Nat, this tees us up for a useful inquiry to just level set for everyone who’s thinking, I still don’t even know what evals mean or what some sort of investigation would include or what incident monitoring would address.

I, full disclosure have been a, a fan of Mackenzie’s analysis here. I now refer to the difference between lawmakers’ expectations and the implementation of that law as the, quote unquote, “Arnold gap”, because he called this out earlier than most folks, and I’m hoping it makes it into some public policy textbooks down the road, but we’ll see.

But Nat, for listeners who are thinking, what would it look like or why do we even need this concept of embedded evaluators, and how would that have addressed, for example, this hugging face OpenAI incident? Can you explain in practice what that could look like from the vantage point of the labs?

Nat Purser: Yeah, so I mean, I think the good thing about the Hugging Face incident is that it did demonstrate the value of outside scrutiny.

I think that like METR and Redwood being able to go in there and get, you know, employee safety personnel level types of access to the systems was very valuable and produced a lot of useful information that legislators and other people have been able to work with. But, I also think it demonstrates the limits of like this kind of episodic approach to auditing or evals, where you go in for a couple days.

I mean, most of the most interesting information that they discovered was like towards the end of their visit, I think within the final two days. So, I think that there are some like clear limits to having people go in for these short stints and bursts, and then do a quick little overview as opposed to something more akin to embedded or continuous assessments.

So, I do think like this was instructive, this... the report that they did, and also demonstrate the limitations of like these current voluntary arrangements. So, I think in terms of like what would be some— a gold standard I know that we’ve talked a little bit about the AEF, the AI Evaluation Forum, their letter with 100 plus signatories, where they call for some minimum standards for organizations within the evaluator universe.

And I think an underrated part of those standards is the editorial independence and the publication rights aspect. People talk a lot about access, and I think access really matters. Obviously, you want the full scope of information necessary to conduct a good audit, but you also need to be able to report on those findings honestly and credibly.

And I think that right now, I can just speak from my own experience working at a auditing organization, the labs have a lot of influence over what goes out when you write a blog post. When you write a report and you say what all you found, they get to go back and forth with you. Their lawyers get to go back and forth with you and say, yeah, we don’t feel comfortable with this being shared.

That shouldn’t be the case. I mean, like with some obvious caveats around like, you know, trade secrets and stuff like that, you just have to have more freedom to talk about the contents of your research. Some other things that we are worried about, which I know AEF covered in their letter, yes, we would love to see more, a more diverse set of evaluators.

I mean, if Accenture wants to take a whack at it, I look forward to seeing what their f- findings look like. I’ve heard good things about the organization that they apparently acquired a while ago. I think it’s called Faculty. And then being able to do like narrow time-limited redactions, protection against retaliation, and then of course, we care about some of these independent standards that lawmakers are starting to think about, such as the auditor not being directly paid by the labs, the auditor not being chosen directly by the labs.

Stuff like that I think matters a lot for like creating a robust network of auditing.

Kevin Frazier: Yeah, and Nat I will claim a little bit of offense to how lawyers purportedly sound. You know- I’m not gonna-

Nat Purser: They have to be part... I understand they have to be part of the conversation.

Kevin Frazier: I won’t hold that against you. I won’t hold that against you.

Nat Purser: Yeah. But I think I would like to do a little bit less interacting with labs’ lawyers if I could.

Kevin Frazier: I’m sure that’s what my students think I sound like. But I, so I do wanna flag one thing that I think is being under-discussed here, and this came up. Nat mentioned Accenture and their new faculty subdivision that they purportedly have acquired and focused on for doing some of this AI eval work.

So, Anthropic announced that they were going to work with Accenture, f- Faculty via Accenture, whatever, and they flagged that, yes, we’re going to pay Faculty for being embedded and doing some of this ongoing continuous analysis. And then at the bottom of the blog post, they mention, and we’re also going to work with METR, a related evaluator, and perhaps other nonprofits.

But we’re not gonna pay them. And, I’m just thinking, why pay Accenture but not pay the nonprofits? And I hope Anthropic clarifies this, but I do wanna stress that if we want more people to be involved in an independent eval ecosystem, then payment should be an expectation. It’s hard to run a nonprofit already.

It’s even harder when you’re allocating all of your staff hours to very intense and very rigorous work. We learned today, for example, that at the UK AC, many folks are feeling burnout because of the stress of this work and the consequences of this work. So, if you’re not paying nonprofits, and if you’re not incentivizing entry into this space, then it’s gonna be really hard to have a robust AI eval ecosystem.

But enough of my soapbox. Alex, I wanna turn it to you.

Alex Bores: No, I would love actually reactions. How do we build a robust ecosystem here? We’re calling for a whole bunch of different IVOs to stand up, some getting paid, some not getting paid, but not getting paid is seen as more independent, but also there’s only a few independent funders of this space, and everyone who seems to get money from a few of those, that’s its own conspiracy.

What is the right way to build out this, this ecosystem?

Nat Purser: Honestly, I’ve gotten a lot of outreach from reporters over the last couple weeks about what embedded evaluations would look like. And when people ask me various questions about like, well, about i- independence, conflicts of interest, whatever, all of those issues pale in comparison to capacity as, like, the issue that worries me the most.

Easily, I mean, right now my, my personal bet here is, and I’ll say it here on the podcast, is that missteps or even just, like, issues with some of these third-party evaluators going in will pave the way/increase interest in government-led or government-supervised audits. I think that it’s— I’ve already seen a lot of Democratic lawmakers speaking to me, that where they’re just like, Nat, why do we want third-party evaluators in there when we could just bring this capacity in-house to the state?

I think that this is complicated. There are federal hiring process reasons. There are competitive pay reasons. I think that, like, m- that this is a maybe questionable model or it, it— the best model might be a complementary thing where you have some audits done by the government, some of it contracted out, or like, we have IPAs with the third-party evaluators.

But I think it— that truthfully, it is quite difficult to build the capacity at scale to do an embedded audit because embedded auditing, when done right, is very labor-intensive. And there are opportunity costs there too, because, like, when METR or Redwood is sending somebody for these months-long or potentially years-long embedded assessments, they’re not doing other things that they could be doing, like creating stronger elicitation techniques or creating more secure evaluation infrastructure.

I mean, they can do that, but they’re reduced capacity. So, I think that, like, it, it’s difficult. We need, like, a government effort. We need certainly a philanthropic effort. We need everybody kind of working in tandem to build out the capacity because it’s critical.

Kevin Frazier: Yeah, Nat, that’s a really good point that even one single incident of a third party missing something, or perhaps there’s a clear conflict of interest where, I don’t know, the leader of some eval org is in a relationship with someone at the lab-

Nat Purser: Yep, is in a relationship is maybe even married to a board member or something.

Kevin Frazier: Yeah. Who could come up with that? You never know what could happen. Who could come up with that? It’s not as though everyone’s a-

Nat Purser: Yeah, but there’s a real, like, principal-agent issues here.

Kevin Frazier: Yes.

Nat Purser: Pe- people not trusting... I mean, we already saw that with, like, the immense backlash to various evaluators from the administration and from other lawmakers of, like, why should I trust these in the blue-haired polycules with my beloved models?

And it’s like, you’re gonna get a lot of that.

Kevin Frazier: I mean, there’s only so many SF house parties that people can intermingle at. So, you’ve gotta have-

Nat Purser: I agree.

 ... some degree of overlap here. And you mentioned the IPA. For listeners who were thinking, oh, I love IPAs- Yeah ... we’re not referring to that kind of IPA.

It’s the Intergovernmental Personnel Act, and it exists at the federal level, which basically says, hey, we want to secund some expertise from an organization, bring them into the government. States have these, too various versions, but some of them are far more restrictive about who can be brought into the government.

And so, to the extent you’re a state lawmaker or state policymaker listening to this and thinking, huh, I wonder how we could bring more expertise into government quickly, check out your state version of the Intergovernmental Personnel Act.

But Mackenzie, with respect to implementation of laws that are increasingly calling for some version of independent verification organizations or some degree of independent third-party auditing, we know that perhaps the biggest laws on the books right now are from California, and Governor Newsom has also not missed the moment with respect to AI being politically salient right now, and issued an executive order basically saying, hey y’all, you know those laws you thought you were gonna have a couple of months to figure out with respect to implementation? We’re gonna speed that all up now and try to really get implementation of SB 813 and related AI laws going sooner rather than later.

Can you walk through why you think that’s occurring and the challenges and opportunities presented by this kind of expedited timeline to try to bring some of these state laws into enforcement sooner rather than later?

Mackenzie Arnold: Yeah, if you have friends in the governor’s office right now, I don’t know, s-send them a DoorDash gift card or flowers-

Nat Purser: Sure.

Mackenzie Arnold: ... or something. They’re gonna need it. It’s gonna be a busy little bit, right? So, so background context, right? We- we’re coming up on a transition between Governor Newsom and the next governor of California.

We only have, you know, four months left on the clock or something like this. We’re now at a point where all of a sudden, we might have a real legislative session at the end of this year, and that’s really the subtext of this EO, right? The, the EO itself can’t change the law. It’s, as one of your former colleagues, Alan Rozenshtein once said, right, it’s an instruction from your boss on really fancy paper, right?

So, this EO is just to sort of get the wheels moving, but the instructions are clear. It is, give us a lot of legislative recommendations that would considerably change what the current state of the law is in California and get them to me fast. Get them by mid-November so that maybe we can act on these.

It moves up, as you said, the timeline for implementing things like SB 813. But it also just shows how quickly the Overton window has shifted, right? So, when SB 813 related to ri- creation of standards and a way of certifying for IVOs in California, when this was created, or the first version was proposed, it was tied to a large liability safe harbor, right?

Then we got version two. Liability safe harbor’s gone. We no longer think that this- Yeah ... is table stakes in order to negotiate what’s in there, but it’s still largely a table setting exercise, right? It’s let’s get ready, let’s seed the market. Maybe someday these will be required. Maybe people will voluntarily have them.

That was signed at the beginning of September. And then a couple of weeks later, now we’re already at, no, okay, accelerate all those timelines, and let’s talk about making those IBOs mandatory. Let’s talk about having some of them be embedded within the companies. Really, this is a huge shift overall. I think it also can address several of the issues that existed with some of the original proposals, right?

So, long as you have your outside auditors be in a voluntary posture, it creates some really bad race to the bottom dynamics, right? It— so long as you are dependent on the good graces and beneficence of OpenAI and Anthropic, you might not wanna push the envelope, right? You might not wanna be fully honest or transparent, and they have incentives to sort of hire whoever is going to be the best box checker out of the exercise.

So, I’m really hopeful about this upcoming period, and I think the sorts of things that you’ll wanna see on those laws go back to what Nat was saying, right? Minimum standards of what does data and system access look like for these people? What do the protections look like such that their disclosures will be truthful and fulsome and are not, you know, redacted to within an inch of their life?

How do you resolve disputes where they come to be between companies and these outside auditors? That’s the sort of stuff we have to figure out, and then you’re gonna wanna wrap that in some sort of flexible structure whereby California can update this over time. We’re not gonna get it right on the first go, but we’re definitely gonna move the ball forward a lot, and then you can re-redo it later on.

Alex Bores: So, we’re sending DoorDash to the governor’s office. The, the outside organizations are— don’t have capacity. Everyone’s running a million miles a minute. Nat already kind of talked about this, but Mackenzie, I would love your take on, does that extend to Congress? Do we think we’re gonna actually get some quick movement here, or is that the one place where people are still...

You know, they don’t need any DoorDash, it’s still gonna be slow?

Mackenzie Arnold: They’re busy all the time, right? They’ve never rested. I think we’ve seen a lot of good movement in the last couple of weeks, right? A- and many others have talked to folks about their incident reporting laws, right, and their internal visibility laws.

I think a lot of them are realizing that the one paragraph that they had on paper before is just not gonna cut it. It’s not what other laws look like when you actually set up an incident reporting regime, not just because you need more words, but because it’s more complicated than that, right? And so, a lot of good work has happened to sort of fix those very obvious holes in things, and that’s honestly been very heartening.

This has happened across the aisle. A lot of people just find this intuitive. A lot of people are interested in getting the right answer to those questions. Is that going to make things move th- this year or next? You’re still constrained by everything that sort of stymies Congress, right? You might see some things get into the NDAA.

Even there, it’s gonna be quite difficult, and you’re just limited, right? This is a, a, a defense spending bill. You can’t park a bus inside of it, though some may try. So, yeah, it’s largely to be seen. I think that the base text that everyone has on the books right now is gonna be much better because of the summer, such that whenever the window does open, we’re gonna get something that makes a lot more sense.

Kevin Frazier: And I think it’s fun to analyze this moment under what some would highlight as the kind of two layers of racing dynamics going on. So, people often ask me, hey, why don’t the labs just stop? And they say, you know, we have a 10% chance of ending humanity within 10 years, so why wouldn’t you stop if you really believe that?

And the kind of common answer is, well, if we stop, then someone who’s less safe is going to rush ahead, and so it’s better that we continue to race ahead such that we’re the ones who win. We doing a lot of work there. Fill in the blank for your favorite lab. So that’s one layer of racing dynamic. The second layer of racing dynamic is, well, if the U.S. slows down, China is only going to rush ahead even further.

And so, we collectively here in the States can’t slow down because we have to beat China. And so, Nat, I wanna come to you because we’re talking on September 22nd. Purportedly, Trump and Xi are going to meet in two days, and AI is apparently on the agenda. We heard from Secretary Bessent that we may have a AI dialogue set up, so some version of an AI red phone for the nations to call back and forth and say, yo, our system got out again. Does this raise a national security concern or not? You should know about it.

For folks who are skeptical of whether Trump and/or Xi will use a red phone, I’m not sure if the red phone has Truth Social capabilities, so that’s one thing that we will want to investigate. But Nat, the idea for some is that, hey, if the U.S. gets its order or its AI laws and orders such that we have transparency and we have some means of verifying the extent to which labs are adhering to these best practices, which I will say for folks who are saying, oh my gosh, this is such a concern, remember that trustworthy AI leads to more AI adoption, which I think a lot of folks claim to be in favor for, and yet a lot of people aren’t using AI because they don’t trust it.

But to put that to a side for a second, how much do you buy into the argument of, all right, if the U.S. does get its house in order, that may facilitate China saying, oh, you know what? We’ll play ball as well. Is there a good chance that we’re seeing movement toward some sort of international or bilateral agreement between the two, or what are you watching these days?

Nat Purser: With the caveat that I’m not a China AI expert my baseline expectation for the meeting coming up is pretty modest. I mean, I think it could result in some continuing dialogue, I think. I mean, yeah, a red phone type thing would be fantastic. Though I think that if you’re calling on the phone, it’s probably too late for whatever you’re trying to do at that point.

But I don’t necessarily expect some kind of like oversight commitments or so certainly not a binding agreement to slow development in a coordinated way. So, I don’t think that I have like super high expectations for this meeting, but I think that like any improvement upon the status quo, any more like regular forms of dialogue would be like welcome and good.

And I think that like, it within the IVO and independent auditor universe, we’re thinking a lot about like cryptographic verification techniques and other ways that don’t rely heavily on like, you know, international trust and coordination to verify that other countries are doing what they say they’re doing within the labs.

But yeah, again I would say overall pretty, pretty modest expectations of what’ll happen here. Would be interested to hear Maclenzie’s take.

Mackenzie Arnold: Yeah. I’m pretty optimistic about this. I think the result might look modest in the short term, but actually be really valuable, right? So, I’m not expecting some grand agreement to come out of this, right?

But U.S.-China relations are this, you know, careful dance played out over a long period of time, and in many ways it’s also... Y- you almost have to think back to olden times where information transfer between the two countries is actually, like, more limited than you might expect, right? They don’t exactly have a model of what the other one’s views on AI and their level of concern is, right?

You’re... and maybe this is my lawyer brain, but there’s something really useful to a meeting of the minds and people coming in and getting a sense for what their negotiating partner is likely to do. And my sense is that after the summer, it- they actually, both countries might feel some bit of kinship in feeling like, oh, we’re actually quite concerned about this, right?

And the concern is not actually each other, it is some, you know, third party, right? It is these companies that are concerning each of us, where, you know, autonomous agents pose a direct threat or, and specific concerns to us and our own sovereignty. And this in some ways puts you on the same side of the bench, right?

It puts you in a posture where both of you are concerned, both of you are maybe willing to signal some amount of willingness to constrain your own market or to take this seriously. And that could actually be really useful, even if it looks really modest and there isn’t some agreement. That might be what it takes to sort of change the posture of each company to sort of get out of the way and be a little bit more comfortable with some of the steps that we have to take on the safety side.

Kevin Frazier: Yeah, and what’s really interesting is folks love to flag the difference in kind of the nature of the predominant concerns in China and the U.S. The, when we’re talking at a high level, the general idea is the U.S. is more concerned about some of these existential risks or perhaps loss of control, whereas China is more focused on the cultural ramifications of AI and perhaps making sure that messaging is controlled.

But in either of those instances, a swarm of AI agents is a problem, right? If your swarm is saying you should buy into democratic liberal Western values in China, I don’t-

Nat Purser: God forbid.

Kevin Frazier: ... yeah, God forbid President Xi, I would welcome that swarm. I’ll put that on the record. ... Please bring your liberal-loving, lowercase L liberal-loving, agents anywhere you want.

But in the same way, as you noted, Mackenzie, that same swarm leading to mass cyber chaos would be a concern here in the U.S. So, I like that focus on swarm... Everyone is kind of anti-swarm right now, or at least anti-swarm, misaligned swarms. But Alex, I’ll turn it over to you.

Alex Bores: Well, I wanna move to the pacing questions but before that, McKenzie, you said you have to imagine this is like olden times. It takes... When is olden times in this analogy? Is this 2021? Is this 1800? Where, what is our channel with here?

Mackenzie Arnold: I’m thinking 1800s.

Alex Bores: Oh, okay. Okay. Right?

Mackenzie Arnold: And maybe this is- it goes beyond just our current governments. This is also, you have to think about the labs, where there’s just constrained information transfer, right? Each of them doesn’t know the negotiating posture of the others.

This is exactly... You know, you create contracts in corporations and other things to make the actions of other people you’re working with predictable, and we don’t right now have good tools for making the actions of either the companies or the countries predictable right here.

Alex Bores: Maybe instead of the red phone, they each need a, a red agent, and we could just have the agents talk to, at superhuman speed on where they are. That might be the next step on, on this negotiation. I wanna move to, to chatting about the pacing the frontier. That, that was really the memetic phrase that went everywhere this summer, and people have different ideas for is pacing a pause?

Is pacing slowing down? How is it verified? Is it happening? It seems like we’re solving Nobel Prize worthy math problems every day. Where is this pacing? Are we on pace to pace? Nat, would love your thoughts.

Nat Purser: Are we on pace to pace? I don’t know about that. I mean, I think that like the, the recent executive order from Newsom like bodes well in the sense of like state lawmakers are stepping up where Congress is taking a step back, or I, you know, we’ll say some leadership maybe are taking a step back.

So, I think the good news is that like California is trying to meet the moment and trying to get ahead of some of the issues that we’re seeing right now. I think that like embedded evaluations will help just like I think... I mean, verification, shared rules, someone empowered to enforce them, all this stuff really matters for pacing conversations, and we’ve got a good starting point here with the company’s safety frameworks.

But, I mean, I feel a little bit bearish on the idea that we could pass a federal framework that will be, that will substantively contribute to pacing in the next two years. I would love to see something akin to, you know, a slightly beefed up and strengthened Frontier Act be passed during this Congress or a, a similar effort but sans preemption, but I think that the odds of this happening are pretty low as long as like the White House is putting out the kinds of tweets that it’s putting out about effective altruism, about AI safety, et cetera, et cetera.

So, I mean, I think that it’s possible that e-either additional warning shots or just the, the state of progression in the California legislature may prompt some more aggressive action. I think that people like Mackenzie and Charlie over at Law AI have raised some important issues around antitrust exemptions, about the possibility that certain AI antitrust laws could impede coordination that needs to happen in order to pace development.

I know that there’s mixed perspectives around that, but I don’t think that as of right now, we’re paced to pace the frontier. I think within the next couple of months as legislatures get into action, that, that could change.

Alex Bores: Let’s go there. Mackenzie, there’s all this talk on potential coordination and what that would mean to slow down.

There was recent reporting in the information that Anthropic and OpenAI almost reached a deal to audit each other’s models using their own models. They just put out a statement a week ago that they wanted to pace the frontier and were immediately hit with an antitrust lawsuit just for saying that they would want to.

What is your take on where these companies can go? Is their real intent to get somewhere themselves or would they need an exemption in order to do any sort of slowdown?

Mackenzie Arnold: Yeah, I think one thing people are missing in a lot of the online back and forth and in conversations on, on, on the Hill about antitrust is that in some ways, which answer is 55% likely of happening just doesn’t matter, right?

What matters is that a lot of very smart people disagree, and a lot of smart people think that there is in fact some amount of risk here. And these companies aren’t angels, right? The, the reason we’re not seeing more unilateral agreements is because they’re unwilling to make those unilateral agreements.

And so, we have multiple tools, and we can pursue both of these tools at the same time, right? One of those tools is the government can intervene, right? They have the authority to come in and make these mandates. Consistent with that, and at the same time, like at any given time, there will be actions that companies could agree to with each other that you have no chance of passing into law, right?

That you like hope someday to put into law. And that, that’s where I really think the antitrust things bites, right? You wanna create a mechanism where those companies feel like, okay, we can make voluntary agreements that are actually costly to us, but are valuable for safety, that we’d be unwilling to make on our own, but we would be willing to make if the agreement was, I’ll do it if you do it.

And as soon as you get into that, I’ll do it if you do it framing, that’s exactly what makes sort of the, the antitrust enforcer ears perk up on this, right? That’s exactly the kind of framing that you’re not allowed to do in most, most cases. Antitrust law is a, a pretty one-note beast, right?

It cares about competition. It does not say, oh, this is valid on other policy grounds. It’s very compelling for society, and therefore we’re going to make an exemption. And courts have been super clear, it’s not their role to make those exemptions, and they’re not going to do it for us. And you know, you read these opinions, and they’re almost a challenge to Congress and says, well, the natural way if you think that a policy reason overcomes what is our natural balance of things, you should make an exemption.

And then the question becomes can you draw that exemption narrowly enough that you don’t create other downstream effects, right? I think that’s doable. I think a lot of people are getting caught up arguing over, you know, w- whether you can come up with some creative workaround here that like actually would at the end of the day win in court.

So long as it’s a creative argument that you have to win in court, it’s probably just that, that, that’s not gonna be good enough, right? We want the certainty now, and I’d like a lot more of this conversation to shift to what do we need to be comfortable with a narrow exemption like that, right?

How can we make sure that it is only limited to certain types of safety coordination, that if it has some alter— like ulterior purpose, that it is going to get thrown out, that the DOJ and the FTC are in a good position to intervene or enjoin some of these agreements if they disagree with them, that maybe some of this information is public or known about these deals, so it’s not happening in a smoke-filled room.

And I think that’s a much more productive direction for this.

Nat Purser: And ve- very quickly, just like, touching on something that Mackenzie said I mean, I remember seeing some people in the antitrust community say stuff to the effect of like, well, you know, catastrophes will reduce output, and therefore agreements that prevent them increase output, and they promote competition, something to that effect.

And I was like, any type of like antitrust maneuvering here that relies on this extremely charitable understanding from a judge where a judge is like, oh yeah, catastrophic risks like decrease like output, and therefore I will, you know, I will not re- regard this as like anti-competitive behavior, I think that is like, that, that’s a very fragile, like, balance that you, that everything hinges on right here.

If it’s all about having an ex-risk pilled judge or something I think that is, that’s a kind of concerning situation to put yourself in.

Kevin Frazier: Well, to Mackenzie’s point and your earlier point, Nat, lawyers are risk-averse, including lawyers at AI labs. And so, if there’s ambiguity, the idea that, oh, but Alvaro Bedoya tweeted that it’s okay, they can coordinate-

Nat Purser: Yeah.

Kevin Frazier: …because there’s a policy guidance statement from 2013 about how companies can share cyber incident reporting and cyber-

Nat Purser: Yeah.

Kevin Frazier: ... concerns. Well, hey, that may have been the case, but I wanna go back even further to real olden times, the 1980s, when we had to issue a antitrust carve-out for the formation of SEMATECH.

For tech law-

Nat Purser: Yeah.

Kevin Frazier: ... nerds who haven’t heard about SEMATECH, this was the idea that we needed to have our best researchers in semiconductors get together here in Austin to share basic research and do basic R&D on that technology. And you only send your best researchers to Austin to do this sort of work if you know you can do it under the cover of the law, and do so in a way that’s going to lead to real returns.

So, I’d welcome a SEMATECH for AI with that carve-out, and yes, you can still come to Austin, and I will still buy everyone breakfast tacos. They’re that cheap. But before I get too repetitive, Mackenzie, we’re running out of time. Any final closing thoughts on your end about what you’re going to be watching as we near the end of September and continue to enter to the fall of AI?

And I’m referring to the season, not that we are entering some sort of winter of AI. But Mackenzie any closing thoughts?

Mackenzie Arnold: Yeah. Two things. One I wanna go back to this point on agent swarms. There’s a chance that how we’ve been thinking about the regulatory object, right, the thing that you’re focused on, which has been so model-centric, is just the wrong target, right?

We’re quickly entering a world in which we’re going to have multi-agent complex systems. Those agents may not always be directed by the same principle. As well, they may interact on the web amongst each other. And this is gonna present all kinds of new policy dilemmas, and I really think that we need to be thinking about that space and moving into it quickly.

The second one goes back to my earlier comments of the dilemma is and always will be uncertainty. Like, people did not predict what the summer would look like accurately six months ago. We will not predict what’s gonna happen six months from now accurately as well. Legislation moves at a pace that is much longer than every six months, and so we’re gonna have to seriously think through how do you make this flexible to update over time?

Some of that’s going to rely on, you know, flexible third-party or voluntary standard setting. Some of it’s gonna rely on capacity that exists outside of government. It— But it’s also inside of government going to have to answer hard questions around rulemaking authority and investigative powers and other things.

And there too, I think the smart space to be moving into is how do you give governments quite a bit of discretion and flexibility, and then make sure that it’s bracketed, make sure that it can’t be abused, it can’t be used for things that would surprise either political party to advance other agendas that they have.

And if we can solve that problem, and it is solvable in words by lawyers, we’ll be in a lot better spot.

Alex Bores: It’s a lot of faith we’re all putting in the lawyers, I have to say. But Nat, take us home. What are your closing thoughts from this conversation?

Nat Purser: I guess my, my overall thoughts right now are that, like, I understand that leadership in the admin may not be as responsive to AI safety issues as we’d like, but I think Congress as a whole has rarely been so generative.

I mean, I just find it very exciting how often staffers are coming to me with, like, really cool, like, novel new proposals and ideas. I think that people have never been so open to kind of experimental or creative legislating. And I also would really enjoy seeing, like, the political factions scrambled a bit, too.

I like that I am working with socialists, I’m working with, like, neocons. I’m working with people who are, like, you know, kind of Joe Rogan-ite, apolitical types. I just find myself working with all kinds of people these days because I think everybody understands the basic issue that we really don’t have much control or oversight over what happens in the labs right now.

And so, I feel very excited about this moment in time. I think that, like, this could be a very fruitful moment legislatively if people, like, really seize the energy of the moment and do something with it. I do think that auditing will, or some kind of independent oversight will have to be a big part of that.

And, but the devil’s in the details, and I really hope that the people both on this call and some of our close friends and allies can help us get those details right. So, yeah, very excited to see what comes in the next couple months.

Kevin Frazier: Well, Nat and Mackenzie, I know everyone has some swarm governance to get off to so I will let you get back to that.

But thank you so much for joining Lawfare. We’ll have you on again soon.

Nat Purser: Thanks so much.

Mackenzie Arnold: Thanks, both.

[Outro]

Kevin Frazier: The Lawfare Podcast is produced by the Lawfare Institute. If you want to support the show and listen ad-free, you can become a Lawfare material supporter at lawfaremedia.org/support. Supporters also get access to special events and other bonus content we don’t share anywhere else.

If you enjoy the podcast, please rate and review us wherever you listen. It really does help. And be sure to check out our other shows, including Scaling Laws, Rational Security, Allies, The Aftermath, and Escalation, our latest Lawfare Presents podcast series about the war in Ukraine. You can also find all of our written work at lawfaremedia.org. The podcast is edited by Jen Patja with audio engineering this episode by me. Our theme song is from Alibi Music. And as always, thanks for listening.