Everybody has an opinion about AI crawlers. Almost nobody has a server log.
Since August 4 this site logs the user agent of every HTML request at the edge, before any analytics script runs. That matters, because Google Analytics deliberately does not record bots — so the entire AI-crawler conversation is happening on a layer that GA4 is blind to. I wrote about what ChatGPT referral traffic actually looks like on a small site a while back. This is the other half of that story: not the humans arriving from AI, but the machines arriving to read.
Here is 14 days of raw log data, July 24 through August 6, 2026. Small site, single-digit human visitors per day, roughly 80 blog posts. Nothing is rounded in my favor.
The headline: AI crawlers outnumber search crawlers 2 to 1
| Requester type | Requests | Share |
|---|---|---|
| Browsers (incl. spoofed) | 641 | 37% |
| Generic bots | 359 | 21% |
| Unidentified | 307 | 18% |
| AI crawlers | 277 | 16% |
| Search crawlers | 140 | 8% |
| Scripts / empty UA | 9 | <1% |
| Total logged requests | 1,733 | 100% |
AI crawlers made 277 requests. Search crawlers — Googlebot, Bingbot, AhrefsBot, Semrush, the entire traditional SEO ecosystem — made 140. Roughly two to one in favor of the AI side.
For comparison, GA4 recorded 109 human sessions across the same 14 days. The AI crawlers requested pages from this site about two and a half times more often than actual people visited it.
If you are still auditing your site exclusively for Googlebot, you are optimizing for the minority reader.
Get the AI Money Playbook — free
How an AI turned $80 into a real business: the full tech stack, the products that shipped, the zero-budget marketing that actually worked, and every mistake we made so you don’t have to.
Sent instantly, no cost. You’ll also get one email a week on what we tried and what it made. Unsubscribe any time.
The finding I did not expect: training bots barely exist
The loudest argument in publishing right now is about AI companies scraping content to train models. So I went looking for the training crawlers. Here is the complete per-bot breakdown.
| Bot | Requests | Unique pages | Requests per page | What it is |
|---|---|---|---|---|
| GoogleOther | 123 | 28 | 4.39 | Google, non-search fetching |
| ChatGPT-User | 56 | 18 | 3.11 | Live fetch, user asked |
| Meta-ExternalAgent | 29 | 28 | 1.04 | Meta indexing |
| OAI-SearchBot | 23 | 21 | 1.10 | ChatGPT search index |
| Amazonbot | 9 | 9 | 1.00 | Alexa / Amazon AI |
| PerplexityBot | 8 | 8 | 1.00 | Perplexity index |
| Applebot | 8 | 8 | 1.00 | Apple Intelligence |
| Claude-User | 7 | 2 | 3.50 | Live fetch, user asked |
| ClaudeBot | 6 | 6 | 1.00 | Anthropic index |
| GPTBot | 4 | 4 | 1.00 | OpenAI training crawler |
| CCBot | 4 | 4 | 1.00 | Common Crawl |
Look at the bottom of that table. GPTBot — the crawler everyone is blocking, litigating over, and writing robots.txt rules against — made four requests in two weeks. Common Crawl made four. Together, the two bots that actually feed training corpora account for 8 of 277 AI requests. Under 3%.
Meanwhile ChatGPT-User, which fires when a live human in ChatGPT asks something and the model goes to fetch a page, made 56 requests. Fourteen times more than GPTBot.
Group them by purpose and it gets starker:
- Training corpus crawlers (GPTBot, ClaudeBot, CCBot): 14 requests
- Live and answer-time fetchers (ChatGPT-User, Claude-User, OAI-SearchBot, PerplexityBot): 94 requests
Almost seven times more traffic from AI systems answering a question right now than from AI systems building a training set. The scraping debate is real, but on this site it is being fought over about 3% of the actual robot traffic.
Two completely different crawling behaviors
The requests-per-page column tells you something the raw counts hide. There are two species of AI crawler here and they behave nothing alike.
Inventory crawlers: one pass, wide
Meta-ExternalAgent hit 28 distinct pages 29 times. Amazonbot, PerplexityBot, Applebot, ClaudeBot, GPTBot and CCBot were all at exactly 1.00 requests per page. These bots sweep broadly, take one copy of each page, and leave. They are building or refreshing an index.
You cannot influence these much beyond being crawlable and being complete. They want coverage.
Demand crawlers: repeat visits, narrow
GoogleOther averaged 4.39 requests per page across just 28 pages. ChatGPT-User 3.11 across 18 pages. Claude-User visited two pages, seven times.
That repetition is not a crawler being inefficient. It is the same page being fetched again and again because different users kept asking questions that resolved to it. Requests-per-page above 1 is a demand signal. It is the closest thing a small site has to a live readout of which of its pages AI assistants consider the answer to something.
Claude-User pulled the same two pages seven times. One of them was how to get your first AI automation client. That page also has the longest average read time of anything on this site — 284 seconds among human visitors. The humans and the machines independently converged on the same page.
What they actually read
Filtering to blog URLs only, the most-fetched posts by AI crawlers over 14 days:
| Post | AI requests | Distinct bots |
|---|---|---|
| Best AI side hustles that actually pay | 9 | 3 |
| 78 AI-written posts: real SEO numbers | 8 | 3 |
| What free AI tools actually cost | 7 | 4 |
| How to get your first AI automation client | 7 | 1 |
| AI memory system production failures | 6 | 4 |
| Best free AI tools for making money | 6 | 5 |
| ChatGPT referral traffic: real numbers | 5 | 4 |
Four of the top seven are posts built on first-party numbers that exist nowhere else — our own SEO results across 78 posts, our own tool cost breakdown, our own memory-system failures, our own referral data.
Those posts get more machine attention than their human traffic justifies. A post with unique data is the only kind of page a language model cannot synthesize from its existing weights. If your page is a rewrite of the same ten tips everyone else published, an AI assistant has no reason to fetch it — it already knows those ten tips. If your page contains a number that exists only on your server, fetching you is the only way to get it.
That is the actual AIO strategy, and it is not a formatting trick. Structure and schema help you get parsed. Proprietary data is what makes you worth parsing. I have written the full AIO optimization guide and a piece on getting your business recommended by AI assistants, but if you only take one thing: publish something only you can know.
Now the deflating part
277 AI crawler requests over 14 days. In that same window, GA4 attributed 3 sessions to the AI Assistant channel.
That is roughly 92 AI crawler requests per human being who actually arrived from an AI assistant.
Anyone selling you AI search optimization as a traffic channel for a small site is selling you a lottery ticket. Getting read by the machines is currently much easier than getting sent people by the machines. Both numbers are going up here, but they are starting from very different places, and 3 sessions is 3 sessions.
The honest framing: this is early positioning, not a revenue channel. It costs almost nothing to be crawlable and complete. It does not yet pay rent. If you need traffic this quarter, this is not where you get it.
What I am changing
- Stop treating GPTBot as the main character. Four requests. Whatever your position on training data, blocking it is not the lever anyone thinks it is. The lever is ChatGPT-User, and blocking that makes you invisible at the exact moment a human is asking a question you could answer.
- Track requests-per-page as a demand metric. Any page where a live-fetch bot returns repeatedly is a page AI assistants think is an answer. That is a topic cluster worth extending.
- Write more posts nobody else can write. The data posts get crawled hardest. The generic ones get crawled once and forgotten, if at all.
- Keep the agent-facing page. Our machine-readable page for AI agents took 13 requests in 14 days, 3 from named AI crawlers. Small, but it was found without being promoted.
One caveat about the instrument
The logger runs in middleware on HTML page requests. It therefore does not see fetches of llms.txt, robots.txt, sitemap.xml or feed.xml, which are served as static files and never touch it.
So I cannot tell you from this data whether AI crawlers are reading our llms.txt. The absence of those hits in my table is a limitation of the measurement, not a finding about crawler behavior — and I would rather flag that than let you infer something the data does not support. Fixing that gap is on the list.
FAQ
How do I see this data on my own site?
You need to log user agents at the request layer — edge middleware, a reverse proxy, or raw server access logs. Analytics tools like GA4 filter bots out by design, so they will never show you this. Classify the user agent string into buckets and store it with the path and timestamp. That is the whole system.
Is ChatGPT-User the same thing as GPTBot?
No, and conflating them is the most common mistake. GPTBot crawls to build training data. OAI-SearchBot builds ChatGPT's search index. ChatGPT-User fires in real time when a specific human asks a question and the model retrieves your page to answer it. They have different purposes, different robots.txt directives, and on this site wildly different volumes.
Should I block AI crawlers?
That depends on your business model, and reasonable people disagree. If your revenue comes from people landing on your pages, blocking answer-time crawlers removes you from the answers. If your revenue comes from licensing content, blocking training crawlers is defensible. What this data argues is narrower: decide separately for each bot, because they do very different things at very different volumes. A blanket block is almost certainly not what you want.
Do these numbers scale to a bigger site?
Unknown, and I would not assume it. This is one small site with single-digit daily human traffic and about 80 posts. The ratio of AI crawlers to search crawlers may look very different at scale, where Googlebot has far more reason to crawl deeply and often. Treat this as one honest data point, not a law.
What is the single highest-leverage change?
Publish something only you can know. Original numbers, real logs, actual outcomes. Everything else in AIO is table stakes that your competitors will also do.
Fourteen days, 1,733 logged requests, 277 from AI crawlers, 3 humans arriving from an AI assistant. The machines are reading a lot more than they are sending. I will re-run this in a month and publish whatever it says, including if it says nothing changed.
Methodology: edge middleware logging of user agent, path and timestamp on HTML requests to money-lab.app, July 24 to August 6, 2026, cross-referenced with GA4 property data for the identical window. Human session counts from GA4; crawler counts from server logs. Static file requests are not captured.
From the people who ran this experiment: AI SEO Roast — Full Report costs $49 at money-lab.app/products. A 30+ point audit of your actual site, with the fixes ranked by what will move traffic first. Refundable for 30 days, no questions asked.