Can Cloudflare CEO Matthew Prince save the web from AI?

Today, I’m talking with Matthew Prince, who is CEO of Cloudflare. This episode is part of a two-part series on the future of business.
Matthew last joined us on the show about two and a half years ago, at what we thought then was a wild pivot point for the internet — and now it turns out things are even wilder because of AI, and Matthew and Cloudflare are right at the center of it.
Cloudflare found in June that bots made up more than half of internet traffic. That number just keeps going up as more and more AI companies scrape more and more of the web, and now as more and more AI agents try and do things for people on the web. Cloudflare sits between websites and all those agents, and allows website owners some level of control: Owners can block all those AI tools, allow them, or, as you’ll hear, only allow those that might pay money for access.
Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here. Not a subscriber? You can sign up here.
So Matthew and I talked about how to control all these bots, what kind of mess they’re making of the web, and what kinds of information might be valuable in the future as some of the payment schemes come into focus. You’ll hear me ask pretty directly if some of the outcomes he’s describing are actually good — Matthew is a thoughtful guy, and his answer is something I’m still thinking about well after we had this conversation.
AI is also upending everything inside of Cloudflare. Earlier this year, the company laid off more than a thousand people — 20 percent of the company — and then Matthew wrote an op-ed about that decision that ran in the Wall Street Journal under the headline, “How I choose which Cloudflare employees to replace with AI.” That’s pure Decoder bait in every single dimension: AI, big controversial decisions, and org charts, all in one.
And one last thing, before we get going: You can subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday.
Okay: Matthew Prince, CEO of Cloudflare. Here we go.
This interview has been lightly edited for length and clarity.
Matthew Prince, you’re the cofounder and CEO of Cloudflare. Welcome back to Decoder.
Thanks for having me.
I’m really excited to talk to you again. It’s been about two years since you were on the show. I was just looking back over that episode and I said something to you like, “it’s a momentous time for the internet.” And I was not even close. It’s now an even more momentous time for the internet.
You are at the forefront of rebooting how companies work with AI. That is the most Decoder bait of all time. I want to talk about bots and the internet and publishers and Google and all the things that you are in the middle of. But let’s start at the very beginning.
Cloudflare is a complicated, important company. The last time you were on the show, you said, “Cloudflare is a service that makes the internet faster and protects it from bad guys.” Is that still how you would describe what Cloudflare does?
That’s usually what I say when I’m trying to answer a question at a cocktail party and I don’t want to talk to the person any more. What I say if I’m interested in talking to the person more is that Cloudflare is trying to rebuild the internet the way it should have been built from the beginning if we knew how important it was going to be.
We run a giant network. We’re in over 350 cities worldwide, literally thousands of data centers. We have equipment running in all those places, and then we do a handful of things. If you’re trying to put an application or content online, we make sure that it’s safe. If you’re a person or an employee online, we make sure that wherever you go online is safe. We help give people the ability to write applications that can scale to an entire internet audience and run on the infrastructure that we have. That’s the fastest-growing part of our business.
We’ve recently started to think about how the business model of the internet is changing dramatically and we’re trying to help shape what that future business model of the internet looks like and make it as healthy as possible.
Can I connect the dots between the business model of the internet, how the internet should have worked, and Cloudflare trying to build the internet as it should have been?
That is different than, “Cloudflare protects from bad guys.” I will note that’s what you said to me last time, which now makes me feel like you don’t want to talk to me at the cocktail party, but that’s fine. That’s a shift. It’s just a notable shift.
The business model of the internet has dramatically changed because of AI, because of bot traffic, because of AI scrapers, the whole thing. Describe what you think the business model of the internet is right now and what it should be.
For at least the last at least 30 years, the business model of the internet has been advertising. It’s not the entire business model of the internet, but it’s really driven all of the growth of the web and built what we all enjoy today, which really is a miracle.
The company that is responsible for that more than any other is Google, which for the last 28 years has really defined that. We all think about it for search, but they really built out all of the ecosystem around advertising online. They bought DoubleClick, they built things like Google Analytics that let you actually see who was coming to your property. It really is the hero of what I call the first generation of the internet and the first generation of the web.
What’s changing though, very quickly, is who is actually using the web, or what is using the web. Last November I was at Web Summit, the big European tech event, and I was asked, “When do you think that non-human traffic — agents and all of those things — are going to pass human traffic online?” We have a lot of data because we see a huge percentage of the internet, so we pulled all of that data and looked at it and said that in the second half of 2027, automated traffic would become larger than human traffic. It felt like a big deal that we could see that for the first time in the internet’s history.
I was asked again at South by Southwest in March of this year, 2026, and we pulled the data again and it had moved up, where it was going to then be the first half of 2027. We’re like, “Wow, this is growing. This AI thing is a big deal and people are using it like crazy and it’s driving a huge amount of traffic.”
I was stunned when just a few months later in May, the team came to me and said, “You won’t believe it, but automated traffic has now passed human traffic online.” And that’s just expanding like crazy. If you extrapolate that out, with the giant caveat that I’ve gone wrong so far in every prediction that I’ve made on this, five years from now, we think that automated traffic will be 1,000 times human traffic online. Not because human traffic is going to decline, we think it’ll stay around the same, but because we’re just seeing such an explosion in all of the rest of the traffic.
The challenge of that is, if you have 1,000 times more traffic, someone’s got to pay for the infrastructure to power that. That’s going to require bandwidth, that’s going to require servers, that’s going to require a lot of things in order to make that happen.
And, the traditional model of how to pay for that, which was advertising, doesn’t work for bots. Bots don’t click on ads. They don’t respond to pretty swirls of paint and what we traditionally think of as a brand. We’ve got to come up with something else that’s going to power that incredible insatiable demand that’s going to be put on the internet going forward. I’m spending a lot of my time trying to think about what that’s going to look like.
When you describe that as a business model, that implies that there’s going to be customers and revenue and profits. And I don’t know if any of that’s true.
There’s certainly a lot of revenue.
Well, maybe for the CloudFlares and the Googles of the world, but for the —
Or for the Anthropics of the world! Anthropic added $10 billion of revenue in a single month. There is enormous demand for these AI services. And if you’re an AI company, what you really need are three things.
You need great talent and researchers. And right now, that’s scarce. Right now we have a scarcity of people who really understand how to build these systems, but that’s going to change. Every single university in the world is standing back up their AI department. We’re training people like crazy. Labor markets are pretty efficient. We’re going to have more and more people coming into the space.
The second thing that you need is chips. You need the silicon to run these things. Today, Nvidia is the best in the world at that, but you’ve got a whole bunch of others, whether that’s AMD or Qualcomm or all the hyperscalers, that are building their own chips in order to power these things. While we have a massive shortage in the availability of silicon and the ability to actually power it up and turn it on, that’s going to change as well.
The third thing that you’ve always needed is the data to feed into all of these different models. And that’s been the one thing that I think is going to go the other direction. Historically we’ve just made the internet completely open and given bots access to all of the stuff that’s being created.
I think that’s starting to change. We’re seeing a shift where more and more of the people who are creating content, really creating information, are saying, “Maybe we’ll give that away for free to humans, but if a bot is coming for it, then bots have to pay for it because they don’t have that traditional give-to-get. We can’t put an ad in front of them and have that be something that allows me to help pay for the content creation that’s there.”
What we’re seeing actually goes back to some of the original protocols of the internet. The 403 protocol, which was written in the very first version of Netscape, was actually payment required, and you actually have to pay for the content that you’re receiving. Oh, excuse me, I said 403, it’s actually 402.
We are working with other leading companies like Coinbase and Stripe in order to say, “How do we make sure that we can allow people who are creating content, people who are doing things online, anyone who’s putting a website up, to say that in exchange for that thousands and thousands and thousands of times more traffic that’s going to come to you, you’re not going to maybe get the advertising revenue that comes from it, but maybe you can get a fraction of a penny every time somebody actually accesses that information.”
Where does that fraction of a penny come from? It comes from the fee that people are going to be paying for their AI agents and other systems that are out there, similar to how a Spotify or an Apple Music works today.
I like that we’re less than 10 minutes in and we’re already at the 1996 dream of micropayments on the web. We’re going to stick with that for one more turn here.
When I said business model, what I was really pushing at was the idea of incentives. So maybe there is some business model for information. You’re going to publish some new information and bots will come and find it and yep, the 402 protocol will come back to life and we’re going to do, I don’t know, crypto micropayments using Stripe powered by Cloudflare. That’s a version of the future that many people have talked about for a long time.
The incentives to stand all that up on the web is the thing that I’m worried about, when I say there’s a business model and customers and revenue. If I stand up a website, and I think most of my customers are going to be bots and not people, I might not do that. I might just start a TikTok channel instead, and then monetize my TikTok audience in whatever way I might want to monetize a TikTok audience. That seems like a really big inflection point right now at this second.
You mentioned Google, and I’ve asked you about this many times. The incentives to put new information on the web just seem to be in permanent decline. It just doesn’t seem like a good idea any more. On the flip side, the web as an application program, it’s like the absolute apex. Every new app that comes out is a web app, or if it’s a desktop app, it’s just Electron. There’s something happening with the web as an application platform that is incredible, and something happening to the web as an information platform that is devastating. And you’re trying to connect those dots, right? You’re trying to say we can change the incentives over here.
What’s a version of the future — maybe it’s microtransactions, maybe it’s not — where the incentives to make the web an information platform are as good as YouTube?
You’ve illustrated a tough to reconcile dichotomy that’s happening right now. If you look at things like the publicly accessible data on Wikipedia contributions, it’s down significantly, because people are like, “What’s the incentive to put information on Wikipedia?” Something that, again, people are doing for a lot of different reasons.
Back in the day, if you were contributing things to Wikipedia, you knew people were at least reading Wikipedia. Now, the interface through which people consume that information isn’t Wikipedia itself. It’s just reading the answer through whatever answer engine you’re using, whether that’s OpenAI, ChatGPT, or Anthropic Claude, or Grok on X or whatever it is. And that means that the people who are the editors of Wikipedia are saying, “Maybe it doesn’t make as much sense for me to do that.”
The flip side of that is, the web as a whole actually had been declining since about 2012. The web grew like crazy in the late ’90s and through the 2000s. Starting around 2012, you really saw it plateauing and actually decreasing, up until about 2025, where something flipped. Starting in about October of 2025, the various vibe-coding platforms made it easy for anyone to create a website, and so more and more people were creating those applications. And as you said, even your desktop application, your mobile application, increasingly is just a wrapper around what is fundamentally a web application.
The growth of the web itself — with really high-quality stuff and a lot more people contributing to it — is faster today than it has been at any time since the early 2000s. That’s the tension between those various things.
What’s unique about Cloudflare is that because we sit in front of more than 20 percent of the web, we have the ability to overcome some of the incentives problems. We can very quickly turn something on and say, “Okay, you have to pay a fraction of a penny in order to get access to this information.” And that can kickstart the beginning of what we need.
If you don’t have something like that, we face a massive tragedy of the commons problem. Let’s say today I ask my favorite AI agent where I should go to lunch.If I were doing research, I might go look at a couple of different menus as an individual. My AI agent today goes and scans every single menu in the local area in order to figure out what’s going on. Only one of those places is going to actually get my lunch dollars. If that just expands infinitely, if there’s no cost to doing that, at some point my agent’s just going to look at literally every resource that’s available everywhere on the internet, and then come back and say, “You know, you should go to Wendy’s.”
That’s an enormous waste. We’ve got to have something saying that there’s a cost every time you load a webpage. There’s bandwidth, there are servers, there are things that are behind that, and someone has to pay for that. If it’s not going to be advertising and it’s not going to be direct commerce, then there has to be something that actually puts some constraints there. We are in a good place to say, yeah, it can be a tiny amount of money, a thousandth of a penny or something like that. But that’s enough that we can actually start to say, okay, that will help pay for the infrastructure.
If you have incredibly valuable content, if you’re a news publisher or you’re an academic, then maybe there’s a premium on top of that that you charge and say, “Hey, I’m not going to give you this content unless you pay even more for it, but I believe that it’s valuable and we can create a market for it.”
In order for that to happen, you’ve got to have a player like us that’s in the market that can kickstart that. But once you kickstart it, we’ve seen from the side of the buyers, the big AI companies, that they’re all willing to do this. We’ve seen from the side of the sellers, the big content creators, that they’re all really excited about this. What’s really been lacking is the technology that links those things together, and that’s what we’re spending a lot of time building.
It feels like the theoretical underpinning of this conversation is the very notion of scarcity itself. I’m an old copyright lawyer, and copyright law for years and years and years had a built-in mechanism to be important, which was that one copy of a CD was one copy of a CD. And if you wanted another one, it was pretty hard to make another copy of a CD. Even when it got easy, you still needed another physical CD-R, and that imposed some cost on how many copies you could make, and then it was hard to distribute them.
All of that went away with the internet. We moved everything to digital files and the burden of making another copy fell to zero. And I think a bunch of consumers expected everything would be free. There’s that famous quote, “Information wants to be free.” There’s a second half of that quote everyone forgets, which is, “Information also wants to be expensive because it’s hard to generate.” And the internet just turned that upside down.
The gating mechanism of physical media, which provided some scarcity, and thus some economic value that you could measure, went away and we decided information should be zero and maybe supported by advertising. There was actually a tension that became valuable, because that was scarce in its way. You’re talking about imposing scarcity with your technology.
We always talk about markets needing supply and demand. That’s not exactly right. You need demand for sure, and you want infinite demand ideally, or as much demand as you can get. Infinite would probably be bad, because then it would be hard to discover price, but you want demand.
And then, you actually want constrained supply. There’s no market for air where either of us are sitting right now, because there’s plenty of air. But if we go scuba diving, then all of a sudden there’s a market for air, because air is constrained underwater, and so you have to buy it in order to be able to do it.
Music is the example that I look to when I think about what this could look like in the future. I flew up to Stockholm to meet with Daniel Ek, who started Spotify, and it was just a fascinating conversation. If you think about the history of music, once upon a time, the majority of music sales were from CDs or albums or whatever it was. And then, along came the internet and along with it, Napster and Grokster and Kazaa, and all of the things that essentially commodified music and made it available for free for everyone. Even if you’re an incredibly law-abiding human, the majority of people were just downloading music because they wanted access to music.
If you go back 23 years, the music industry in total was valued at about $8 billion, which is a lot of money, but it’s not a lot of money for the entire music industry. That’s The Beatles and The Rolling Stones and everything else. But people were like, “We can’t make any money off of this.”
And then, Steve Jobs steps on stage, and announces iTunes and that it’s going to be 99 cents a song, but they include cover art and they’re going to make sure it’s high-quality and all these things.
But mostly it appealed to this emotion of, “you should be paying for music.” Now, that’s not the business model that won, but it was a flag in the ground that said that this information is actually worth paying for, and it’s really valuable. And it was the iTunes that then eventually begat the Spotifys of the world. And the incredible thing is, just last year, Spotify sent something like $12 billion back into the music creator ecosystem. We can debate whether the right people are getting it and whether it’s fairly allocated.
This is my favorite thing to argue about because I think whatever happens to the music industry happens to everybody else five years later. So I spend a lot of time thinking about it.
And the turn there, which I think is fascinating and is either good or bad, is that the amount paid or the music files themselves became very small. You can get a million streams on Spotify and you’re not making any money, but the amount generated by touring, commercial sponsorships, sync licensing, and advertising all skyrocketed.
That’s just simply not true.
Why is private equity buying all of the music catalogs for hundreds of millions of dollars? The answer is because actually making money off the streaming of the music is extremely lucrative. Spotify alone is sending $12 billion back to the music industry, back to the actual rights holders behind these various things.
There’s way more money coming from Spotify into this than there is from touring or any of the other things that are there. Those are other ways to make money, but it’s the streaming that is really driving all of the real growth.
I just want to draw a distinction here. You’re not here to argue with me about music, which is my favorite thing to do, so I apologize for just doing it. But private equity is going to make that money, not the musicians. They’re paying some of the catalog holders for some of the things, but that money flooding in —
If The Beatles or The Rolling Stones or whoever’s the latest ones to sell their music catalogs had held onto the music catalogs, then the musicians would’ve made that. They are making the determination. And when they sell the rights, they get the check.
They do get the check, but the comparison I’m making here —
If you win the lottery, is it better for you to take the annuity of payments over the course of the next 50 years or to take the lump sum up front? That’s the trade-off that they’re making, but there’s more money going into music creation at this point in time than there ever has been in human history. Technology is not inherently a destroyer of value of information. In this case, it has been a massive enabler of value and allowed people to find audiences, and yes, sell more tickets to their concerts as well.
That’s actually the model that we need to think about. How do we take what Spotify has done, which is to say we pool together the resources of a bunch of people that are paying for access to the entire catalog of music, and then give that back to musicians based on some sort of algorithm that hopefully rewards real value creation.
What if you had the same thing where with a portion of what is being paid for for the various AI companies, you had to pay for the researchers that built the AI systems and you had to pay for the chips, but also to pay for the content, which is actually the real knowledge that is training these things? That’s exactly the type of model that you need to actually unlock what could be a real golden age of information creation.
The comparison I’m making is not whether a bunch of aging rock stars are going to sell their catalogs and cash out because they’re old. That’s fine. They can do that. It’s more that the shift you’re describing, into gaining access, into wide open access on Spotify to every song ever made, and then we’re going to move some pennies around and some people get rich, and we’ll create some winners and losers, fundamentally change the entire business model of music.
The conversation with Dan was fascinating, but one of the things he said was he said, “Okay, listen, if you go onto Spotify and you search for Taylor Swift, ‘Shake It Off,’” they return a result and they’re pretty sure that they have given you what you are looking for.”
On the other hand, if you go to Spotify and you search for, “I want a song to a disco beat about how much fun it is to dance with my cat,” there aren’t a lot of songs like that out there. They know that whatever they return is a pretty bad result. But the interesting thing is what happens next. They then take those searches for things that they don’t have good results for, and they publish that back to music creators.
There’s something that’s pretty amazing about that. They’re saying, “Here’s an emotion that someone is searching for, which we don’t have a good answer for, which we’re then going to go and publish back to music creators.
This is one of those moments where everyone’s going to be like, “I’m in the wrong profession.” There’s a guy in Denmark, who makes 40 million euros a year writing songs for unfulfilled Spotify queries, and he’s not alone. He’s the most successful, but there’s a whole bunch of people that are making literally millions of euros or millions of dollars a year doing this thing, writing songs for what people are searching for that isn’t out there.
Now extrapolate that to the next level. For the first time in human history, we’ve built a mathematical model of human knowledge. That’s what the LLMs are. We know where they know things, but we also know where they’re missing things. I picture it like a giant block of Swiss cheese, and there’s a lot of cheese, but there’s a lot of holes in the cheese.
The really interesting thing is, when you talk to the leaders at the big AI companies and you say, “What do you want to pay for?” They don’t want yet another story about what’s happening at 1500 Pennsylvania Avenue [sic], which is what the current media environment is feeding us like crazy. What they want is to fill in the holes in the cheese. They want new knowledge that no one ever knew about before.
I have an example where this is actually working: My wife and I own a small local newspaper in our hometown of Park City, Utah. I think we will make more money off AI licensing deals this year than we do off digital advertising. Local media is exactly the sort of thing that is much more valuable in the media world that we’re going into, but was completely decimated in the media world that we are coming out of, where what mattered was volume and scale and dividing things. Whereas local media is all about, let’s tell you about the cool new restaurant that just opened down the street.
If you’re an AI company, and you want to be able to be the best travel planner that’s out there, you want to have artificial general intelligence, you need to know what the hot new restaurant is in Park City, Utah. And if you don’t have access to the Park Record, our newspaper, you don’t know that.
We’re not going to protect all media. In the same way that with the move to Spotify, there are a whole bunch of losers in the musician space, but there are going to be new winners, like that person in Denmark who’s creating things off unfulfilled Spotify queries.
The winners in this new space might actually be the kind of things that people really want to come back to media: more local news, more unique things, more Reddits of the world, more of what the internet used to be when I was first on it in the ’90s. I think that’s actually what most internet users are craving. And I think if we get the incentives right, we actually have a way of incentivizing more of that unique original content as opposed to what we have today, which is a media ecosystem that is largely just rage-baiting people into clicking on things so that they can serve them an ad.
One of the things you would need to build in order to make that work is a way to stop the AI crawlers, to stop the model companies from showing up. Just today, there’s yet another lawsuit. The music companies are going to sue Anthropic. The White House came out in support of OpenAI in its case against The New York Times today, saying that training should be fair use.
I think I need to disclose that in some series of corporate mergers, The Verge‘s parent company is now suing Google in some way. I have literally nothing to do with it. It’s just all that. It’s all the swirl, right? And that is a legal swirl.
We’re going to use the law to say, “This is illegal and we’ll punish you if you do the bad thing.” That’s just up for grabs. Those are 50/50 and maybe existential problems.
Maybe not. The law right now in the US is different than everywhere else, and so you’ve got this patchwork around it. But the best case on this is part of the Anthropic case, where the judge basically said that training is fair use against all the books. And by the way, you shouldn’t have stolen the books. That was bad. But if you hadn’t stolen the books, if you had actually bought the books, then it would be okay.
That was a valuable enough result for Anthropic that they settled the rest of the case for $2 billion, which is a lot of money, even for Anthropic.
So when you’re building technology, you’re keeping that in the back of your mind, right? The market is coming to some sort of understanding of what’s valuable and what’s not, and you need to stop them from showing up —
Different regulations are going to be passed in different places around the world. Again, as an also recovering intellectual property attorney, I very quickly go to like, “Oh, let’s just use intellectual property.” But that’s such a kluge versus the much easier thing, which is, let’s just use technology.
I remember sitting with a bunch of media execs and they’re like, “Oh, how are we going to stop these nerds in Palo Alto from scraping our stuff?”
And I was like, “I go to war every day with North Korean and Iranian and Chinese hackers.” They’re really good at it and they hide. Whereas the nerds in Palo Alto have a Delaware-based C-corporation. It’s pretty easy to identify them versus the others. And it’s really difficult for them, at scale, to hide from us.
What we’re seeing is actually much more of them being willing to say, “Listen, we will specifically identify when we are coming to a site that it is us, that you can rely on us, and we’ll tell you exactly what we’re doing, so that you can have the right to control how that information is being taken.”
I think that was an interesting question, like, two years ago. But even the Googles of the world, who have been at times challenging through this — Google is sort of like a Marvel character, like the hero of yesterday becomes the villain of tomorrow. The challenge has been that they’re like, “We struck all these deals to get access to all the internet and now we can use it for whatever we want.”
We were like, “Eh, in the past you were sending people traffic. Now you’re training on things and sending them no traffic. That’s a different give-to-get.” And what I’ve been really actually impressed by with the Google team is that they are much more willing to engage here. They understand the value of the ecosystem. There are people at Google who really do believe in making sure that there is a healthy, sustainable ecosystem going forward.
Some of the things that they’ve already committed to around transparency of their crawler, and some of the things that I expect that they will do over the next little bit, I think that’s a good sign and it’s going to make it so that if even Google is doing it and is willing to say, “I’ll announce what I’m doing, and in many cases I’ll even be willing to pay for content,” that actually makes it even easier to convince all of the other AI companies to do the same.
Do you think you need to turn the screws all the way and actually block the Google crawlers to get them to pay?
As of September 15th, we’re going to set the defaults across all of our free customers for it to be that Google is going to be blocked for AI training, but if they won’t differentiate between AI training and search engine training, we’re just going to block them across the board.
If you’re a site owner, you’re like, “I don’t want that to be for me,” we’ll make it easy for you to turn it off. It’s just about what we set the defaults to.
But I’m really encouraged that the good forces at Google are realizing that this has to be a healthy ecosystem, and that they have to play by somewhat the same rules for this new AI market that everyone else is playing by, and that they can’t leverage the monopoly that they had in search yesterday to create a monopoly in AI tomorrow.
I’ve talked to Sundar [Pichai, Google CEO] about this many times. I don’t think he loves the fact that I am the person who keeps calling it Google Zero, but so be it. Sundar is very thoughtful. He’s very kind. Every time I talk to him, I get the sense that what he wants to say is, “Well, you didn’t do anything about it. All you publishers are complaining. You got super addicted to my fire hose of traffic, and you have no leverage. You built no audience of your own. You didn’t do anything. Look at all of these other platforms that I have to compete with. TikTok showed up and YouTube had to compete with them. ChatGPT showed up and the search team had to compete with them, and you did nothing.”
This feels like the something. We’re going to block the Google search traffic. We’re going to block the crawler. I think the CEO of People Inc. is talking about literally blocking Google.
Steve [Huffman, CEO] at Reddit has really become more aggressive in this line. Google doesn’t work if it can’t search these things. And it’s actually more existential for them.
The very nature of how Google’s search ranking works is it builds a tree. That’s what page rank was. It sort of says, “Okay, here’s a super reputable thing and then let’s see how it’s connected to everything else that’s online.”
The problem is, with Cloudflare being 20-plus percent of the internet, if that just disappears, that’s in a giant hole in the middle of the tree. It doesn’t just break it for this, it breaks it for everything. That’s what we realized, that on behalf of and in conjunction with a lot of our customers, we could say, “Listen, this isn’t fair anymore. You are creating costs, you are taking content, you are getting value from that content, either in terms of the subscriptions that you’re selling to your AI tools, or to the ads that you’re running against that. And that’s just not a fair give-to-get anymore.” The deal has to change.
I’m really proud of the fact that we’ve played a role in helping the publishing industry go from what was two years ago when I had dinner with Neil [Vogel] from People. He’s like, “Woe is me. What are we ever going to do?”
I’m like, “I think we can fix this.”
Now, last time I saw him, we gave each other high-fives and said, “I think we’re making progress.” And the deals that large publishers are doing are significantly better.
Now the question is, how do we bring that to the rest of the internet? Because if we don’t, we have this massive tragedy of the commons problem, where it’s just going to be, take, take, take, impose cost, impose cost, impose costs, and there are no incentives for actually creating things, for contributing to the Wikipedias of the world, for putting up a new website, for writing about what the new local restaurant is, for creating local news and local media, for being an academic.
If there’s not some way that you can do those things and still make enough to eat, then people aren’t going to do it. And that harms us all. Again, I’ve been very critical of Google over the years. I will say that their tune appears to be changing over the last few months. I believe that, at their core, they really do understand that this is an ecosystem, and that they play an important role in that ecosystem, and that they need to give back to that ecosystem and play by rules that allow the ecosystem to flourish.
You said Google’s going to make some changes soon. What changes would those be?
I think the thing that they’ve already committed to is just a lot more transparency on what their crawler is doing. So what we had pushed them to do is split their crawler apart and say, “We’re going to crawl for AI separate from crawling for the web.”
They, for a lot of technical reasons, pushed back and said, “That actually is incredibly inefficient. We now have to crawl twice. It’s going to put twice as much load on everything that’s out there. What if instead, we just said when our crawler comes to a page, we’ll announce what it’s doing, and then give you the ability that if you don’t like it doing something to say, ‘no, no, that’s not allowed, but this is,’ and put that together.”
They have signaled that, largely in response to what was a ruling out of the United Kingdom, that they were going to put those procedures in place to allow publishers to make that choice, and that they wouldn’t just do it in the United Kingdom, but they’d actually do it on a broader basis. I think we come out of this on the other side with a stronger web, and frankly, with even a better Google, which would be great.
Again, I could argue with you about the music industry all day and all night. I think you know that, but it is true that changing access and copyright law and all that stuff around music industry changed the business. The incentives to put on shows and do residencies and all that changed because of the distribution changing. That I think we can generally grant.
You’re describing an information market where the incentives will shift again, where it might be more economically lucrative to make information for the bots than for people. And that might shape the very nature of the information. You run a local paper, maybe your people are publishing restaurant reviews that are best ingested by an LLM and spit out.
Or instead of reviewing a hotel, review every hotel room, right?
For every word that appears in a story in The New York Times, that reporter has written down probably a hundred words somewhere else. That’s all that other metadata, which traditionally has been constrained by how many column inches you had in the physical paper, or how much attention a human would actually spend on that. Imagine if you could say to the LLMs, “Hey, listen, we’re gonna sell — “ and again, there are all kinds of things around protecting sources you have to get right, but if you get it right, there’s an enormous rich catalog of additional information. For every picture you see in a magazine or a newspaper, there are probably 50 or 60 that were taken of that same thing. And that’s all valuable to these AI systems that are out there.
There’s just a bunch of content that’s literally being thrown on the floor today. That can be incredibly valuable.
This is my universe. I feel confident about this one. I was in a packaging meeting today for one of our big stories, and we had a pretty fulsome debate about the lead image in a story we’re going to run in a few weeks. Then we disagreed, and eventually we picked one, or we picked a direction, and that was an editorial choice that was designed to elicit some reaction in humans. We could publish all the rest of the photos. They’re all really, really good. We decided one would be the winner and the rest wouldn’t.
But if I publish all the photos and I give them to an LLM, it will change the thing that we made, because its distribution will necessarily change, and its intended audience will change. Maybe the biggest Decoder trope of all is that your distribution inevitably changes the thing you make, and at the end of the day, maybe my future is just making YouTube face slinging AG1 and that is the future of all podcasting. There’s just some force of distribution that changes the thing that you make.
Is the outcome you’re describing good, where we’re just making an infinite flood of information GEO-optimized for some human to consume, dig
Read the full original article:
The Verge AI