All posts by Matthew Conroy

The Internet has a second audience

Post Syndicated from Matthew Conroy original https://blog.cloudflare.com/agentic-web/

For most of its history, the Internet had one audience that paid the bills: people. We read the articles, saw the ads, and bought the subscriptions. Bots were always there, but they were mostly large, automated operations that didn't view ads, pay for anything, or read in any meaningful sense.

That's changing fast. At the end of 2024, Cloudflare handled an average of 63 million HTTP requests a second. Today, it's almost doubled to 115 million, with peaks above 150 million. Over the past year, daily requests from AI agents on our network grew by more than 1,700%. This year, for the first time, more than half of Internet traffic wasn't human.

The human web didn't shrink to make room. A second audience arrived alongside it: agents, software acting on behalf of people. They sit somewhere between humans and traditional bots. They don't respond to ads, but there's usually a person behind them with a job to get done. For businesses that learn to serve them and capture value from them, agents are additive. For those that don't, they're extractive.

What our customers need hasn't changed: to be discovered, to tell great stories, to build great experiences, and to sell. What's changed is that more than half your visitors are now software. Our job is to help you serve both audiences.

More traffic, less revenue

For thirty years, the web ran on one arrangement: you let search engines crawl your site, they sent you visitors, and you turned those visitors into a business. Being found and getting paid were the same thing.

AI has caused this delicate balance to break down. Now, answer engines read the page and give the reader a summary. This costs websites bandwidth without leading a human to a website where the ads or payments happen. The machines kept coming, and the audience that paid for the web stopped reaching those sites. Some of the most heavily crawled categories, like Retail, Computer Software, IT & Services, and Financial Services, have seen human traffic decline as much as 40% in less than one year.

The result is that revenue per request is falling while costs are rising. Every automated request still costs bandwidth, compute, and origin capacity, and a growing share of those requests carry no referral, no ad impression, and no subscription. Our first instinct was to block all automated traffic. Last year we recommended blocking AI training crawlers on new domains so site owners could at least say no to their content being used to build models. In Spring 2025, 22% of the crawler requests we saw were for AI training (according to the crawlers’ stated purpose). By June 2026, it was 52%. The problem is a blanket “no” is not a sufficiently nuanced approach for the Internet economy being built right now.

The opportunity is there to cater to agents. Get it right, and you are at the forefront of a new business model. Get it wrong, however, and the results will be the same as they were for generations of websites that were on the wrong side of search engine algorithm changes.

Some of that traffic is a customer

An agent booking a table, comparing insurance quotes, or buying a dataset for a researcher is a customer. It just isn't a human one.

The fastest-growing part of automated traffic is no longer crawlers. It's agents: software fetching pages on a person's behalf, often because the human asked a chatbot something. That agent traffic follows human routines, with a weekly rhythm and a dip over the summer holidays. Turn an agent away, and you may be turning away the person who sent it.

Agents also behave differently from training crawlers. A training crawler collects your pages to build a model. An agent comes back each time someone asks about that content, so this traffic grows with how many questions people ask, not how much you publish.

You can't do business with an audience you can't see, can't tell apart, can't set terms for, and can't charge. Until recently, for most of the web's non-human traffic, none of those four things were possible.

See who’s really visiting

"AI bot" no longer means anything useful. What matters is what a bot does. Cloudflare’s AI Crawl Control, Business Insights, and BotBase show site owners who is crawling, what they take, what comes back, and which of your URLs they want most.

A bot's name is only worth something if you can trust it. With Web Bot Auth, operators, including OpenAI, Google, and AWS, cryptographically sign their agents' requests, so a site can tell a real agent from an impersonator without guessing from IP addresses or user-agent strings. We see more than 500 billion verified bot requests each week. 

Set your terms

In July, we replaced the single "block AI bots" switch with separate Search, Agent, and Training controls, available on every plan, including Free. The data showed why that distinction was needed. Fewer than 1% of sites on Cloudflare block search crawlers, while 17% block training. Site owners were never trying to hide. But with the rise of agentic traffic and the new ways agents use information, they suddenly had no transparency into, or choice over, how their content was being used. Being found no longer ensures they get paid, and they want to be found without being exploited.

That’s particularly difficult in the case of mixed-use crawlers. When one bot does both search and training, refusing one means refusing the other. On September 15, we shipped Disallow AI Training. It keeps you indexed for search while using crawler-specific mechanisms to instruct the operator not to use your data for training. Apple, Google, and Microsoft have committed to honor it. Cloudflare Radar also publicly tracks crawler behavior.

New domains now see recommended configurations based on how the site makes money rather than what piece of software is visiting. For ad-supported sites, you can easily disallow training and block agents on pages that carry ads, because an ad only pays when a person sees it. You can change any of these settings at any time.

Get paid

In August 2026, we described the Agentic Internet we're building as readable, discoverable, callable, and payable. The last word, payable, is the one that determines whether the open web can fund itself. The web needs a way to say ‘yes, if you pay’ instead of a binary ‘yes’ or ‘no’.

The licensing market shows both how much demand there is and where the gaps are. More than 50 publisher-AI deals have been signed since 2023. Nearly all of them are bespoke and bilateral, between large publishers and large AI companies. They prove content has value. But they don't reach most of the web, and they don't reach most buyers.

Not every asset should be sold the same way. High-value content and datasets need a trusted network, where buyers are identified and report how the work was used. Services like APIs and MCP tools don’t work like that: every request is the use.

So we're building for both.

Pay Per Use reaches the sites that direct licensing can't. Most publishers will never get a bespoke deal with each AI company, and no AI company can negotiate with millions of sites. Pay Per Use is the bridge. It doesn't charge for the crawl. It pays when content is actually used. Every buyer is a verified crawler, which is what makes this a trusted network, and each one defines what counts as use and what it will pay.

Publishers see the offer, choose whether to opt in, and are able to opt out whenever it stops working for them. The buyer reports each use, Cloudflare checks those reports, then bills the buyer and pays the publisher. The reporting matters as much as the payment. Publishers see what was used, when and what they earned, and, where the buyer reports it, information about which questions surfaced their work. Licensing deals rarely show any of that. It creates a feedback loop: publishers learn what people are actually asking for, and from that can decide what to cover, what to update, and what to make readily available to agents.

There won't be one definition of use. A search engine citing a source, a research agent quoting a passage, and a shopping agent completing a purchase create different kinds of value, and each will want its own business model. Buyers can participate via multiple business models using the same rails, with no new integration for publishers. Take a trade journal for marine engineers, with a few thousand subscribers and little prospect of an AI licensing deal. It gets paid by every participating AI company that draws on its work.

Monetization Gateway captures value that has never had a way to change hands. Accounts, API keys, and subscriptions work for customers you already know, not for an agent that wants one lookup from a service it has never used before. Our closed beta allows eligible U.S. Cloudflare customers to put a price on anything that passes through us, using the Rules language they already know. When a rule matches, we return an HTTP 402 Payment Required using the open x402 protocol, and the agent pays the seller directly.

That does more than recover lost revenue. Agents are customers in their own right: they pay for the data, APIs, and tools they use, whether the request is the whole purchase or one step in a larger task.

Monetization Gateway prices per request, per query, or per token, at fixed or capped prices. A sports statistics site built on ads can charge a fraction of a cent each time an agent asks "who leads the league in assists?" When we announced Monetization Gateway, thousands of sellers joined the waitlist, and their most common request was "charge agents, not humans." We're also our own first customer. Cloudflare's AI Gateway uses Monetization Gateway to let agents pay for inference, so we find the rough edges before our customers do.

For buyers, both products beat a block page: reliable access, and a way to reach millions of sites instead of one licensing deal or API key at a time. Every paid request leaves a receipt showing what was bought and that it was paid for.

Both Pay Per Use and Monetization Gateway are bets, built with customers on shared primitives: identity, metering, pricing, settlement, and analytics. They work together, so a publisher can disallow training, allow search, earn from AI answers, and charge agents per article from one dashboard. Pricing and discovery aren't solved yet, which is why both launch as betas, shaped by real customers and real transactions.

Make every request cheaper

Payment is the answer to falling revenue. Rising cost is a different problem, and much of it is simply waste. Most crawlers still download pages built for humans, again and again, to extract a few paragraphs of text. Too often, bots crawl sites that haven’t changed since the last attempt. That burns bandwidth for the site and compute for the crawler, and it happens before any answer is written. We’re working with our customers and the crawlers on tools that will help. Today, you can see the bandwidth consumption used per operator in our dashboard.

In July, we announced a joint research project with OpenAI, a first-of-its-kind pilot to explore how insights from Cloudflare’s global network can help AI search engines discover and index relevant content on the open web more efficiently and effectively. We’re planning to share our initial results in the next few weeks.

For our customers, we’re shipping tools and one-click experiences to make their sites optimized for this new kind of traffic. Markdown for Agents lets agents read a page without the additional styling meant for human eyes, and WebMCP lets a site expose actions directly instead of making agents guess which button to press.

Why build on Cloudflare

More than 20% of the web sits behind Cloudflare’s network, and so do nearly 80% of leading AI companies. We see both sides of this market. We build the rails for visibility, identity, controls, and settlement, and let the market work out what things are worth.

The old deal is gone, and the new one is still being written. Together we can shape what happens next.

In one version, a few companies control how agents find things, prove who they are and pay, and everyone else routes through them. In the other, those pieces are open standards anyone can implement, and a site of any size can set its terms and get paid. We prefer the latter.

That's why these rails run on open standards like x402 and Web Bot Auth, so anyone can build on them. Domain owners choose their own identity providers, their own payment processors, their own agent partners. Cloudflare is one option, not the whole stack.

For decades, the web was paid for by the people who visited it. Now the software visiting on their behalf can pay its share too.

From ranking to recommended: get your site ready to thrive in the age of AI agents

Post Syndicated from Matthew Conroy original https://blog.cloudflare.com/aeo/

Your next customer may not find you through a search engine. Instead, they'll ask an AI assistant: "how do I do X?"; "which option is best for someone like me?"; "just handle it for me" and an agent will find the answer, weigh the options, and act on their behalf. Increasingly, the moment that determines whether a customer chooses you happens inside a model's response — before a human ever sees your homepage.

This agentic audience is already here: by our count, fewer than half of all HTML page requests now come from a human. Not all of those machines are agents acting for a person, but that share is growing fast, and answer engines, shopping assistants, and research tools will shape which businesses are found and recommended. Discoverability used to mean ranking on a results page. Now it means being found, read, and confidently recommended by the agents that guide your customers.

The old metrics, human clicks and page views, no longer paint the full picture. We spent time talking to site owners who were staring at access logs full of AI bots, completely blind to whether those bots were capable of using their site or recommending their products and services to their users. We heard two main questions:

  • Can agents actually use my site?
  • Am I getting recommended?

To help site owners answer these questions, we have integrated our previous work on Agent Readiness into the Cloudflare dashboard, and added our new Answer Engine Optimization (AEO) tool as well. These tools treat agents as a core user base for your site, showing you how an agent will see it, and how often you get recommended.

The opportunity is big, and the bar is low, because most sites aren't built for this user yet. Just as early SEO rewarded the sites built for search engines, the sites built for agents will be rewarded now. The ones that are easy to find, read, and trust are the ones agents will recommend.

Diagnostics: is your site ready for agents?

Diagnostics is the technical checkup within Agent Readiness. It scans your site the way an agent reads it: it works out whether it's allowed in and whether it can discover your content, fetches a clean machine-readable copy, and finds the interfaces it can call. 

While a person just loads your homepage, an agent leans on your robots.txt, your sitemap, your response headers, a Markdown version of your content, and published metadata for authentication and tools.

Diagnostics runs those checks against a hostname and rolls the results into a single agent-readiness view, from "Not Ready" to fully agent-native. Every check comes back as pass, fail, or neutral, with a note on why it matters, and an evidence trail showing the exact request and response we saw.

The checks are grouped by effort, so you know where to start:

  • Quick wins: the high-impact basics most sites are missing, including a crawler-readable robots.txt, an XML sitemap, AI-crawler rules, and serving clean Markdown to agents
  • Technical groundwork: the next layer, including Content Signals that state how your content may be used, an API catalog, link headers, and agent login instructions
  • Advanced integration: the agent-native features, including OAuth discovery, MCP (Model Context Protocol) and A2A (Agent2Agent) agent cards, a skills index, Web Bot Auth, and WebMCP
  • Commerce: the emerging agent-payment standards including x402 (an extension of the classic HTTP 402 Payment Required status code), ACP (Agent Commerce Protocol), Universal Commerce Protocol (UCP), and AP2 (Agent Payments Protocol). This is informational for now, and not counted in your score.

Every suggested improvement comes with a next step. When there’s a Cloudflare feature that can help, there's a "Set up in Cloudflare" link straight to the setting, such as switching on Markdown for Agents or managed robots.txt. For everything else, there’s a "Copy Agent Prompt" button that proposes what your coding agent needs to build. Make the change, re-scan, and watch the checkmark turn green.

AEO: are AI assistants recommending you?

Diagnostics tells you whether agents can read your site. The AEO tab tells you what happens next: when a customer asks an AI assistant a question in your category, does it recommend you or a competitor? You can't look this up like a search ranking. There's no impression count and no missed-click report, so when a competitor gets named instead of you, the sale is gone and nothing tells you it happened.

We infer your industry (e.g. health and fitness) and category (e.g. sports apparel) from your site, and we probe the leading assistants (today, Anthropic's Claude and OpenAI's GPT) with likely customer prompts to see how they respond. We structure these prompts to mimic real-world discovery, asking for recommendations, product comparisons, and general advice within your category. By observing how models answer these realistic queries, you get metrics such as:

  • Citation Rate: the share of answers in your category that cite your site as a source
  • Prominence: when you are cited, how much of the answer is actually yours and how early it lands
  • Mention Rate: how often assistants name your brand in their answer — for example, how often "Cloudflare" shows up in the response, whether or not cloudflare.com is cited as a source. Read alongside your Citation Rate, it separates awareness from attribution: assistants naming you far more than they cite you means you're on their radar but not yet earning the citation — a specific, targetable gap.
  • Share of Voice: your slice of citations against those for your competitors, so you can see who is winning the prompts you're losing

To evaluate how an AI model perceives your market presence, we build a benchmark across each industry and category before scoring a specific site. We query AI assistants with likely prompts in that category — without specifying your brand — and record which sites are cited, where they appear, and how prominently they feature.

Rather than re-querying models every time a site owner runs a scan, we run this panel once per category and reuse the baseline across all accounts in that domain. Pre-computing this dataset provides three main benefits:

  • Zero latency: Results load instantly from a snapshot rather than waiting for live model queries.
  • Lower compute overhead: Aggregating queries at the category level avoids redundant AI calls across thousands of scans.
  • Industry Fit scoring: Reusing the panel corpus lets us map which brands consistently appear together, allowing us to derive an Industry Fit score that measures whether an AI assistant views your site alongside your actual competitors.

AI assistants rarely answer the same question the exact same way twice. To account for this variance, we use Cloudflare AI Gateway to prompt each assistant multiple times across different models. We then read the responses a customer would see — the answer text alongside the sources each assistant cited — and extract multiple signals from it. 

We evaluate not just whether your site was mentioned, but whether you were cited as a source, how early your citations appear in the answer, and how much of the final answer's substance is attributed to you. Where genuine judgment is required, Workers AI does the heavy lifting, running natively on our own infrastructure to read each reply and score how your citations and mentions appear. We also use exact text analysis rather than a model grading its own output. Together, this folds dozens of one-off replies into actionable metrics. By abstracting the multimodel query and evaluation pipeline, the tool provides metrics without requiring you to build your own evaluation framework.

Alongside the answers, an AI Operator Activity shows the real crawl and referral traffic on your site, per operator (OpenAI, Google, and so on): who reads your content, who sends visitors back, and the errors they hit on the way (403 blocked, 404 dead link). The pattern worth acting on is the operator that crawls thousands of your pages but refers no one, using your work without sending customers back.

Because these numbers are specific to your site, you can experiment, re-run the scan, and measure the impact on the exact questions that bring you business.

Meet your other audience

Until now, sizing up agents meant guesswork: grepping your logs to infer who visited, or feeding a chatbot a prompt and eyeballing whether it mentioned you. But with Agent Readiness and AEO, you can get the data you need to act. And because the requests actually pass through Cloudflare, these tools measure rather than estimate where possible, and will improve over time. 

Helping you see who's reaching your site and decide how to engage on your own terms is what we've always done. Agents are just the newest audience, and the businesses that make themselves easy for agents to find, understand, and trust are the ones that get recommended. Agent Readiness is where you find out whether you're one of them, and what to do if you're not yet.

Ready to find out if AI agents are sending customers your way? Head over to the Overview tab in your dashboard to get your site Agent Ready and request early access to AEO Visibility.

Building on the open, agent-ready web? Open the Agent Readiness tab in your Cloudflare dashboard and tell us what you're building on the Cloudflare Developer Discord.

Making AI search smarter

Post Syndicated from Matthew Conroy original https://blog.cloudflare.com/making-ai-search-smarter/

Search drives most experiences on the web. It’s how we get things done, and how nearly everything on the web gets found — the creators, the merchants, the answer to whatever you just typed into a box. For nearly 30 years, that discovery journey ran on a simple bargain: let a search engine crawl your content, and it sends you visitors. You turned those visitors into a business — through ads, subscriptions, or just the audience itself. Being discoverable and getting paid were the same thing. A year ago, on the first Content Independence Day, we drew a line to defend that bargain in the AI era. But a line in the sand was only a first step. Since then, the prevalence of AI search in consumers’ lives has only accelerated as more than 50% of traffic online is non-human. The threat is no longer a handful of training crawlers you can block; it’s search itself being rebuilt around AI answers.

Today’s answer engines read your page and hand the user a summary, so the visit — and the revenue that depended on it — isn’t needed. We see it firsthand, and independent research backs it up: a 2025 Pew Research Center study found that when Google shows an AI summary, users clicked on a traditional search result link just 8% of the time (about half as often as when there’s no summary) and clicked a link inside the summary only 1% of the time. That leaves our customers in a bind: opt out of AI and be hard to find, or opt in and deliver significant value to users while seeing increasingly little in return. Our customers want to be found and compensated for the value they provide, and right now they’re forced to choose.

Today, we’ve announced new bot options to help our customers better control who can access their site and what they can do with it. But blocking was only step one: saying “no” protects content without rebuilding the business models that sustain it. So, it’s time to start building the new economic model of the Internet, starting with search.

Rebuilding the bargain

Transparency and control are the foundation, but more is needed. In 2025, we laid out our foundation via a set of responsible AI bot principles: bots should be transparent about who they are and what they’re for, respect site owners’ choices, and act in good faith. Our tools hold bots to that bar. But enforcing good bot behavior doesn’t make AI search any better for the people relying on it, and it doesn’t send a dollar back to the creator whose work made the answer possible. We can do more than help the web say “no”; we can help rebuild what it says “yes” to.

So today, we’re announcing two initiatives that move from defense to offense and start putting both halves of that old bargain back together.

Make AI search smarter: By using the signals we see across our global network, like what’s fresh, what’s high quality, and what’s actually changed, we can help answer engines surface the most relevant content and reduce unwanted crawling. People searching get better answers, while costs are reduced for both AI companies and site owners if webpages are only recrawled when they’ve changed.

Pay creators for the value they provide: When your work is used to answer someone’s question, you should be rewarded instead of just being scraped for free. And you should be able to see what’s being used and what people are asking. This should be a real revenue stream, and an incentive to keep producing original content worth finding.

Making search smarter

Today we’re launching a research program to make AI search smarter and stop our customers footing the bill for crawls that produce nothing new.

More than 20% of the web sits behind Cloudflare’s network, which gives us a unique perspective. We can tell which pages have genuinely changed and which ones people and agents are flocking to. Through this program, we will explore using signals our customers have chosen to share about the freshness of their content, and we will combine those with our own insight into traffic flows, both human and bot. For answer engines, that’s a roadmap to high-quality content. For our customers, it provides a view of what users are actually asking, and how their content shows up in AI results. The aim is to measure two things: how much these signals help answer engines to surface fresher, higher-quality content, and how much unnecessary crawling they cut out.

That second benefit, cutting unnecessary crawling, is bigger than it sounds. Cloudflare data suggests that more than 50% of crawl traffic from good bots goes to re-fetching pages that haven’t changed — and that number is likely to climb as crawl volumes do. A signal that just says “nothing’s changed here” lets a crawler skip the trip. That saves the answer engine compute. More importantly, it saves site owners from serving and paying for requests they never needed to. 

The program is neutral by design: our goal is to make it work for every answer engine willing to play fair. It’s limited to search. We aren’t sharing any content, and nothing is used to train foundation models. We intend to publish what we learn, including the benefits to site owners such as better content discoverability and reduced server strain. We plan to make the capability broadly available later this year and reduce unnecessary crawling across our network.

From Pay Per Crawl to Pay Per Use

Last year we launched Pay Per Crawl so publishers could charge AI companies for crawling their content. It was a real start, but crawling is a crude measure of value. A single page might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all. Creators want to be paid fairly for the value they provide.

So we’re starting to shape Pay Per Crawl into Pay Per Use. We’re running experiments with top AI companies, like Ceramic.ai and You.com, and the arrangement is straightforward: organizations can bring their payment models and easily scale them to content owners across the Cloudflare network.

Ceramic has built what it calls a “pay-per-query” model, so publishers who opt in can be paid when their content appears in Ceramic’s search results. This means payment is designed to follow the value the work delivers rather than the number of times a crawler happens to fetch it.

“To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation,” says Anna Patterson, founder and CEO of Ceramic.ai. “Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results.”

In addition to compensation, content owners participating in the Cloudflare/Ceramic program will unlock new reporting to help with answer engine optimization (AEO). Customers can finally see the top queries leading to their content appearing in search results, the specific webpage and snippet, their average search result ranking position, and more. This is the first of many products we’ll be launching to help our customers with discoverability.

This is just one emerging approach. Another comes from You.com: agents can pay on demand for a specific piece of premium content they need, without any upfront commitment. New payment models from AI providers are being tested (e.g., Pay per Query, Pay per Result, etc.) and we have the infrastructure to support them all. 

We want to be honest that this is an experiment. There’s a lot to learn, including exactly how this holds up at the scale of the Internet. We’ll work that out with our partners and our customers as we go, and share what we learn. But the goal is clear: AI search companies get fresher, better-grounded answers, and the customers whose work makes the answers possible get paid when they help. Cloudflare’s job in all of this is to provide the infrastructure layer that makes this market flourish. 

We think this is a more natural fit for where the economics of search are heading. The old, human web optimized search to save time — providing excerpts, ten blue links, and a click. The agentic Internet is different: an agent can read fast and search continuously. Search is becoming something an agent does dozens of times to answer a single question, closer to a utility than a destination. In that world, the unit that matters isn’t the crawl or the click. It’s the outcome. Pricing the outcome, and paying the people who made it possible, is how the web continues to thrive.

The headline we want to earn

A year ago on Content Independence Day, the headline was a default ‘no’: AI can’t crawl without compensation. This year, our focus is on giving our users more products and controls to say ‘yes’ and bring more benefits with it.

Today’s announcements are just the beginning. Cloudflare’s research project is designed to see if our signals produce better results with less crawling. Pay Per Use is a promising direction we’ll experiment with alongside partners who believe that content creators deserve fair compensation for their work. This is how the last 30 years of the web got built too: somebody runs the pilot that turns “the model is broken” into “here’s the new model,” one experiment at a time. We believe there’s value to our customers to be discoverable in this new agentic era, and to optimize their content for maximum discovery. But they should be able to do this without giving away their most valuable creative assets for free.

The web is changing, and the business models it’s relied on are changing with it. The old Internet was open, neutral, and worth contributing to. We have a rare chance to keep it that way, and to build the business models that fund it in the future. Smarter answers for humans and agents asking the questions. A fair deal for the people whose skill, creativity, and commitment makes the answers worthwhile. That’s how we pursue Cloudflare’s mission: to help build a better Internet.

Happy Content Independence Day!

Building on the open, agent-ready web? If you are interested in learning more about the Ceramic and You programs, please fill out this form. If you’re building an answer engine and want to crawl smarter, we’d love to hear from you too: [email protected].