The Bots Ate the Web. Now Cloudflare Wants Them to Pay.

(SeaPRwire) –

By: Oliver Hawthorne

The sixty-percent mark arrived in May, eight months ahead of the timeline Cloudflare’s CEO had projected. Matthew Prince expected bot and AI agent traffic to surpass human web traffic by the end of next year. It happened faster. Almost 60% of all requests to websites across the internet now come from automated systems, not people. At the current burn rate, Prince projects the ratio could reach 1,000-to-1 within five years. That is not a rounding error. It is a structural inversion of the internet’s fundamental consumer. The machines now outnumber the humans accessing the web. The anxiety here is not about SEO rankings or search visibility. It is about extraction without return. AI crawlers ingest content to train models. Those models may never redirect a reader back to the source. Not a cent of revenue. Not a single click. Prince spoke from his office at One World Trade in downtown New York. He expected the crossover point at the end of the year. It happened in the spring. That acceleration tells you something about how fast AI companies are scaling their training pipelines. The timeline compression is the real story. The internet’s machine layer did not creep into dominance. It sprinted. The shift was supposed to take twelve more months. It took zero. The five-year window to reach that 1,000x ratio is not theoretical. The trajectory is already in motion.

Cloudflare manages traffic for more than one-fifth of the entire web. Eighty percent of the top AI companies are among its customers. The company has a unique vantage point. It sees every bot request, every crawler signature, every traffic pattern flowing through its infrastructure. This week, Cloudflare launched a “Disallow AI Training” setting. It lets a website remain indexed by search engine crawlers while blocking AI training crawlers from scraping its content. The technical distinction matters. Publishers can still get organic search traffic without feeding the training pipeline. The crawlers that bring readers are kept. The crawlers that train models without sending anyone back are blocked. Apple, Google, and Microsoft have all agreed to honor this separation. That is significant. These three companies control the search infrastructure that most of the web depends on. Their buy-in means the “Disallow AI Training” mechanism is not just a theoretical toggle. The agreement from Apple, Google, and Microsoft means publishers can now make a choice. They can keep search visibility. They can block AI training. They no longer have to choose one or the other. In parallel, Sony and Warner Music have filed copyright suits. News outlets and even major dictionary publishers are pursuing legal action against AI companies over intellectual property claims. The “Disallow AI Training” tool is not a replacement for litigation. It is the first operational lever publishers actually control. It runs through existing web infrastructure rather than through courtrooms. The legal path is slow. The traffic management path is immediate. This is also one of the first real mechanisms publishers have to separate “index me” visitors from “train on me” crawlers. Before Cloudflare’s announcement, publishers had no practical way to make this distinction at scale. They could block all crawlers, which would kill their search traffic. Or they could allow everything, which would mean free content extraction. There was no middle ground. Now there is. The setting runs through Cloudflare’s existing infrastructure. Publishers do not need new software. They do not need legal counsel. They flip a switch.

Prince’s argument is direct. “Companies need to get paid by the AI companies for what is the fuel that runs these AI systems,” he said. He also flagged something less obvious. “I’ve been really focused on: How do we make sure people get paid? But I worry that I don’t think we’ve thought enough about: How do we make sure that people get recognized?” Writers, filmmakers, and musicians value credit as much as compensation. Anonymity and intellectual theft cut deeper than unpaid labor. Prince sees a gap between the payment question and the recognition question. He believes there should be an Academy Award or Nobel Prize of knowledge to give credit where it is due. He also drew a broader line about the media’s evolution. “The media world in the last 30 years really rewarded popularity and I think that the media world of the next 30 years might really reward credibility,” he said. That observation lands differently when paired with the 60-percent figure. A web where machines outnumber humans by 1,000-to-1 is a web where algorithmic noise dominates. Credibility becomes the only durable asset. Prince envisions a future with 500,000 AI companies rather than five. He acknowledges the prerequisite for that scale: a payment mechanism for content creators. The gap between five AI companies and 500,000 is not just a number. It is the distance between a centralized extractive model and a distributed, paid model. “If we don’t have a way of paying content creators, of funding the infrastructure buildout, of making everything more efficient, I just don’t know what the internet looks like in the future,” he said. Without monetization, the infrastructure buildout cannot sustain itself. The commercial loop is currently broken. Content creation has real costs. AI training delivers real benefits. The extraction is free. If Cloudflare can weaponize its position as the traffic management layer for two-fifths of the web, the economics shift. Publishers who organize around the “Disallow AI Training” setting gain leverage. AI companies that refuse to pay face a shrinking pool of usable training data. Attribution and compensation are critical for building sustainable media models that reward labor, creativity, and the cost of producing work. That holds whether the content is artistic, scientific, or journalistic. Bots can spit out millions of articles that riff off the news. Someone has to report what is really going on. The companies that win this transition will be those that negotiate the compensation framework first. They need to move before the crawler ratio reaches 1,000x.

Author bio: Oliver Hawthorne, a Principal Correspondent stationed at an international technology review, covering infrastructure shifts, platform economics, and the AI-content monetization gap across global digital infrastructure.