How Much Bandwidth Do AI Crawlers Cost? Work It Out From Your Own CDN and Origin Logs

The bandwidth line on your bill jumped a tier. The site wasn't redesigned, ad spend didn't move, and the first suspect is AI crawlers. The problem is proving it. A lot of teams get stuck in exactly the same place: they know AI crawler traffic is up, but they can't say how much bandwidth it actually eats — and they certainly can't point to the line on the invoice it lands on.

This article won't hand you an industry average, and it won't give you a reference range. The reason is blunt: the cost of AI crawler bandwidth depends heavily on how many long-tail pages you have, how your caching is configured, and how big your images are. Change any one of those variables and the answer shifts by an order of magnitude. Rather than copying a number that looks authoritative but has nothing to do with your site, take the measurement method instead.

Six things follow: why the cost lands on origin fetches rather than CPU, who is actually crawling you (with the state of four official IP lists as measured on 2026-08-05), which five log fields you have to keep, the four steps that convert raw logs into origin GB, how to match that against your bill to find the line item you're really being charged for, and why we recommend setting rate limits first instead of blocking everything at once.

Caption: The bill gets expensive at exactly those requests that slip past the edge cache and go straight back to the origin — the origin fetch is where the meter runs.

The Cost of AI Crawlers Isn't CPU — It's Origin Bandwidth and Cache Misses

The conclusion first: when AI crawlers make your bill more expensive, the cause is usually not that your servers can't keep up. It's that the crawler is hitting places your CDN edge can't cache, forcing origin fetch after origin fetch. Ordinary visitors and crawlers simply want different things — the former cluster on a handful of popular pages, the latter wants everything.

Visitor access is a spiky distribution. The homepage, a few popular articles, a few product pages — those absorb the overwhelming majority of traffic. Those pages run hot at the edge, hit rates are high, and the origin barely has to lift a finger. That's exactly where a CDN's cost savings come from; for the mechanics, see our explainer on how CDN edge nodes and caching actually work.

Crawler access is a flat distribution. It doesn't favor popular pages; it walks your sitemap or link graph top to bottom, including the long-tail pages nobody has clicked in three years. Long-tail pages have no heat at the edge, which means they aren't cached, which means nearly every fetch turns into an origin request. Ten thousand requests from visitors might wake the origin a few hundred times; ten thousand requests from a crawler might wake it every single time.

Build the Skeleton of the Estimate From Page Count

To estimate the order of magnitude for your own site, the skeleton is just three multipliers:

1. Long-tail page count — how many pages on your site "could get crawled but nobody normally reads"
2. Fetches per pass — how many times the same set of URLs gets crawled within one billing cycle
3. Origin bytes per page — the HTML plus the images and static files pulled alongside it on first load

Multiply the three and you have the origin volume of one complete crawl pass. The first number you can look up right now; the other two only come out of your logs.

Take CloudInsight's own site as the example: as measured on 2026-08-05, 365 articles in Chinese and 369 in English, 734 in total. That's not a large content site, but it demonstrates the point — when one crawler walks all 734 URLs end to end and most of them are cold at the edge, the origin gets woken up 734 times. If your site has five thousand pages, or fifty thousand, substitute your own numbers and look at the result.

We're deliberately leaving the second and third multipliers blank. "How many passes" and "how big per page" can differ by more than a factor of three between two sites in the same industry — any number we filled in for you would be fiction.

Why It Isn't a CPU Problem

Crawler requests are mostly GETs for static content. They don't trigger logins, don't touch the shopping cart, don't run complex queries. Unless your origin has to run a full dynamic render or a database query every time it responds, CPU usually holds up fine — bandwidth and request count are what start hurting first. So why does everyone check CPU first? Because it's the most prominent thing on the monitoring dashboard. You look at it, decide "that's fine," and miss the item that actually got expensive.

Caption: Left = ordinary visitors concentrated on a few popular pages (high edge hit rate). Right = crawler access spread almost evenly across the whole site (heavy origin traffic from long-tail pages).

Start by Listing Who's Crawling: Four Official IP Lists and How Often They Change

To assign the cost you first have to know whose account it goes on. And "AI crawler" was never one thing — when Cloudflare introduced its AI bot categories it pulled out ten of them at once, including Search Engine Crawler, AI Crawler, Page Preview and more. The taxonomy alone tells you something: crawlers for training, crawlers for search, and crawlers triggered by a user pressing a button behave completely differently, so their cost structures differ too.

The good news is that both OpenAI and Anthropic publish the egress IP lists for their own crawlers, and that's the first raw material for grouping. Below is the state of play as measured on 2026-08-05 — and note that these values move, so you should re-fetch them yourself while reading this.

Crawler	Purpose	Official IP list	Prefixes	List creationTime
GPTBot	Training	https://openai.com/gptbot.json	21	2025-10-30
OAI-SearchBot	Search	https://openai.com/searchbot.json	35	2026-01-02
ChatGPT-User	User-triggered	https://openai.com/chatgpt-user.json	289–290	2026-08-05 01:03
ClaudeBot / Claude-User / Claude-SearchBot (one shared list)	Training / user-triggered / search	https://claude.com/crawling/bots.json	20 (IPv4 20, IPv6 0; 19 of them are single-IP /32s)	2026-05-01

(Figures above are first-hand measurements from 2026-08-05 and will change as the vendors update their lists.)

Re-running it yourself is quick — one line does it:

# 2026-08-05 實測：讀出清單的建立時間與網段數
curl -s https://openai.com/gptbot.json | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print(d['creationTime'], len(d['prefixes']))"
# 輸出：2025-10-30T11:00:00.000000 21

The Update Cadence Reveals Two Very Different Maintenance Costs

Line up the creationTime values from all four lists and a pattern surfaces (this is our analysis, not an official statement): training crawlers have extremely stable egress IPs, while user-triggered crawlers churn constantly.

The creation time on GPTBot's list sits at 2025-10-30 — untouched for nine months as of 2026-08-05. ChatGPT-User is the exact opposite: its creationTime was 2026-08-05 01:03, the very day we measured, and when we called it twice a few minutes apart the prefix count dropped from 290 to 289. Same day, same script, minutes apart, different answer.

The difference in scale is just as stark: ChatGPT-User's 289 prefixes are roughly fourteen times GPTBot's 21. The two vendors also style their lists differently — 19 of Anthropic's 20 entries are single-IP /32s, whereas OpenAI leans on /24 and /25 blocks. That has a direct effect on what it costs you to hand-maintain WAF rules: maintaining 21 stable rules is a different job from chasing 289 prefixes that drift.

A Trap We Hit in Testing: the Official IP List Is Itself Behind Bot Protection

This one is worth calling out separately, because it will make your automation fail silently. Same https://claude.com/crawling/bots.json, change nothing but the User-Agent, and the response code changes:

User-Agent	HTTP response
Python-urllib/3.13	403
curl/8.7.1	200
A normal browser UA (Mozilla/5.0 ... Chrome/128.0 Safari/537.36)	200

(First-hand measurement, 2026-08-05)

Three commands reproduce it:

# 2026-08-05 實測：同一個 URL，只換 User-Agent
curl -s -o /dev/null -w "%{http_code}\n" -A "Python-urllib/3.13" https://claude.com/crawling/bots.json
# → 403
curl -s -o /dev/null -w "%{http_code}\n" https://claude.com/crawling/bots.json
# → 200（curl/8.7.1 的預設 UA）
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36" \
  https://claude.com/crawling/bots.json
# → 200

The first time we wrote this sync script, we opened the URL straight from the Python standard library, got a 403 — and nothing blew up. The script quietly wrote out an empty list. If you plan to schedule an automatic sync of these lists to feed a WAF allowlist, remember two things: set a respectable User-Agent explicitly, and make the script raise on a result like "prefix count is 0" instead of letting it silently overwrite yesterday's good data.

This Step Is Labeling, Not Identity Verification

To be clear about what you're getting: matching User-Agent strings against official IP lists gives you grouping, not identity verification. Anyone can forge a UA string, and an IP list can only prove that a source falls inside the officially published range — it can't prove nobody slipped in by another route. Verification in the real sense involves reverse DNS, cryptographic signatures and a separate set of mechanisms; we cover that in the official methods for verifying AI crawler identity.

What does the industry bar look like? Cloudflare's Verified bots documentation (the page is marked Last updated 2026-07-01) lists two criteria. First, honest self-identification, achieved through Web Bot Auth cryptographic signatures, a published IP list paired with a stable User-Agent, or reverse DNS. Second, non-abusive behavior — respecting robots.txt, keeping request rates reasonable, and not circumventing the site owner's wishes. The reverse DNS (FCrDNS) that Google uses to verify Googlebot is one of the veteran approaches to the first criterion.

Conversely, Cloudflare's fake bot managed rules flag a request as a fake bot when "the UA matches a known bot but the source can't be verified." Which makes the point neatly — a UA is a claim, not evidence.

For the traffic-costing exercise, labeling is precise enough. But if your next move is to turn that list into blocking rules, insufficient precision converts directly into collateral damage.

Calculating AI Crawler Traffic From CDN and Origin Logs: Five Fields, Four Steps

How do you actually look up crawler traffic? Not in the pretty charts on your monitoring dashboard — in the raw logs. And the first obstacle for most teams isn't that they can't do the math; it's that they never kept the fields they needed in the first place.

The Five Fields You Have to Keep

Field	Why it's non-negotiable	What breaks without it
timestamp	Slices by billing cycle, exposes crawl cadence and peak windows	You get a total, but no idea who hammered you and when
user-agent	The first layer of grouping	No way at all to attribute traffic to a specific crawler
client IP	Cross-checks the UA label against official lists	You can only trust the UA string, and spoofed traffic counts toward your total
response bytes	This is what the bandwidth bill is actually priced on	You can only count requests, and request count correlates weakly with money
cache status	Separates edge hits from origin fetches	You can't tell which requests actually disturbed the origin, so the origin-fetch ratio is uncomputable

The last two are the ones most often missing. Without response bytes you'll unconsciously use request count as a proxy for cost — but does it proxy well? A 2 MB hero image and a 30 KB HTML page are both one request, and the gap on your bill is more than sixty-fold. Without cache status you can't even answer "did that fetch hit the origin at all" — and the origin fetch is the very thing this article sets out to measure.

Four Steps to Convert Logs Into Origin GB

Step	Action	Output	Easiest thing to get wrong
1	Label and group by user-agent	Every request tagged with a bot name or "general traffic"	Loose keyword matching that sweeps in unrelated UAs
2	Cross-check against official IP lists	A count of suspicious entries where the UA claims a bot but the IP isn't on the list	Comparing against a list you pulled months ago, misclassifying a whole batch
3	Split by cache status into edge hits and origin fetches	Origin request count and edge hit rate per bot	Every CDN names its status values differently; a bad mapping table ruins the split
4	Sum response bytes across the origin-fetch subset	Origin GB per bot for that cycle	Including edge-hit bytes in the sum and inflating the total

Run those four steps and you'll hold one table: each bot, its origin GB, its edge hit rate. That table is the basis for every decision that follows — without it, any rate limit or block is pure instinct.

Three Common Ways to Get the Math Wrong

First, counting requests instead of bytes. This is the most widespread one, because request counts come ready-made on most dashboards while bytes have to be dug out. The result is that you treat a pile of HTML fetches as a crisis while missing the images and static assets that are actually eating your bandwidth.

Second, looking at the edge and ignoring origin. That total traffic figure on the CDN dashboard — how much of it actually disturbed the origin? A substantial share may have been served straight from the edge with the origin never involved. Those two things are priced differently on your bill, and blending them means putting your effort in the wrong place.

Third, log retention that's too short. Keep only seven days and you can't produce "this month versus last month," nor line up with a billing cycle measured in months. If your site also has to absorb real-time requests from AI agents, retention needs to run longer still before patterns emerge — we go into that separately in getting your servers ready before AI agent traffic arrives.

---

Want to See What Your Traffic and Billing Actually Look Like First?

Most teams stall at step one: log retention is too short, or cache status was never captured, so the origin-fetch ratio can't be calculated. The CloudInsight technical team can help you inventory the traffic and billing structure of your cloud and CDN setup, so you can see which line item the money is going to.

👉 Contact our technical team

---

Turning Traffic Into Money: Finding the Line Item You're Actually Billed For

Once you have origin GB, the next question is which box on the invoice those GB land in. Billing dimensions differ across CDN and cloud providers, but crawler traffic usually shows up in the following places.

Common billing dimension	Where AI crawler traffic lands	What drives the amount
Origin / egress traffic (GB)	The main battlefield. Every fetch of a long-tail page may be an origin fetch	Long-tail page count, page weight, edge cache hit rate
Request count	Crawlers are inherently request-heavy, but the unit price is usually far below traffic	Number of crawl passes, static assets attached to each page
WAF / Bot Management rule evaluations	A cost that only appears once you start blocking	How granular the rules are, how often they're evaluated
Log export and storage	Incurred by doing the very analysis in this article	Retention length, export frequency, storage tier

Here's the counterintuitive part: the log export you turn on in order to understand crawler cost is itself a cost. That's not an argument against turning it on — it's a reminder to count it, and to decide how long retention should be once you have your answer, rather than leaving everything on indefinitely. To line up billing items across providers, see our full breakdown of how CDN pricing is calculated.

Why This Article Doesn't Give You an "Industry Average"

By this point you may still want a number: how much do AI crawlers eat at other companies? Roughly what share of total traffic?

We're not giving one, and that's deliberate. For such a number to mean anything, the site you're comparing against would have to resemble yours in long-tail page count, caching strategy and page weight. Get any one of those three badly wrong and the answer moves by an order of magnitude. Handing you an authoritative-looking range only invites you to treat it as a baseline for decisions, and then spend effort in the wrong direction.

The only credible number is the one on your own bill. The four conversion steps in this article aren't a theoretical exercise; they exist to get you your own number. Once you have it, you're in a position to argue about whether to act, and which lever to pull.

Why We Recommend Rate Limiting Before Blocking: Three Side Effects That Bite Back

Should you block AI crawlers? Before you have your own numbers, the question has no answer. After you have them, our recommendation is still: set rate limits first, and treat a full block as the last resort. Three reasons.

First, blocking by IP may stop you from communicating your wishes at all. Anthropic's own documentation is blunt about it: blocking by IP may not achieve opt-out correctly or durably, because it prevents the crawler from reading your robots.txt. That sentence is worth a second of thought — what you consider "blocked" may register on the other side as "unable to read your statement of intent." You've walled off the very channel that expresses refusal.

The standard spells the consequence out more concretely. RFC 9309 section 2.3.1.3 states that if a crawler receives a 4xx status code when fetching robots.txt (403 being one such code), then "the crawler MAY access any resources on the server" — it may treat the entire site as carrying no restrictions at all. Block robots.txt along with everything else via IP rules, and the instruction the standard gives the other side isn't "stay away," it's "help yourself." The opposite of what you intended.

Second, a blocklist costs more to maintain than a rate rule. Look back at the 2026-08-05 measurements: ChatGPT-User's list was updated that day, and between two calls minutes apart the prefix count went from 290 to 289. How often would you have to sync a list that drifts like that to keep up? A rate rule, by contrast, is behavior-driven — whatever IP the other side moves to, exceeding the threshold gets throttled, and you never have to chase a list.

Third, a bandwidth invoice shouldn't get to decide the visibility trade-off on its own. That's a content-side question, and no cost spreadsheet can answer it. AI SEO Hacker, run by the same team as CloudInsight, has a dedicated piece on what blocking AI crawlers does to your search visibility; this article only handles the cost half.

Dimension	Rate limiting	Outright blocking
Effect on cost	Flattens the peaks, total stays under control	Zero in theory; in practice depends on how precise the rules are
Maintenance cost	Write the rule once, it follows behavior	Requires syncing against IP lists that drift
Collateral damage risk	Throttles only above the threshold; observable and adjustable	An expired or over-broad list blocks entire groups
Effect on opt-out	robots.txt stays readable, your wishes get through	May prevent the other side from reading robots.txt
Reversibility	Change a parameter, effective in minutes	Requires unblocking and waiting for trust to rebuild

If you want to know how thoroughly AI crawlers currently read your site, and what you'd lose by blocking them, first run through the checklist for an AI crawler accessibility audit and capture the current state before you touch anything — otherwise you'll have no control group after the change.

Caption: Left = rate limiting (flow narrowed but the channel stays open, so your wishes still get through). Right = outright blocking (the channel is welded shut and even robots.txt becomes unreadable).

How to Set the Limits: Three Throttling Levers and Their Side Effects

Crawler rate limiting isn't a single technique. You actually have three levers, and they differ in effect, risk and effort.

Lever	How it works	Side effect	Recommended order
Loosen cache TTL for bot traffic	Lets the edge serve requests that used to hit the origin	Longer visible delay before content updates appear	Do this first, lowest risk
CDN / WAF rate rules	Throttles or challenges above a threshold	Too tight a threshold catches legitimate crawling and real visitors	Do this second, controllable and reversible
Crawl-delay	Declares your desired crawl interval in robots.txt	Advisory, not enforceable; the other side may ignore it	A supplement, never the main measure

Lever One: Loosen Cache TTL, the Only Lever That Cuts Cost Without Cutting Crawls

This layer is especially worth doing first, because its logic differs entirely from the other two: it doesn't try to make crawlers fetch less, it makes each fetch stop disturbing the origin. Origin fetches become edge hits, cost falls naturally, and the crawler still gets the full content.

The price is freshness. Can you afford it? Extend the TTL and updated content takes longer to become visible at the edge. For long-tail content like spec sheets, documentation and archived articles, that price is usually acceptable; for a price page you edit daily or anything real-time, be careful. For which parameters to change and how to measure whether hit rate actually improved, work through our hands-on guide to cache strategy and hit-rate tuning.

Worth noting: this lever only works if you confirmed a high origin-fetch ratio during the four-step conversion. If your crawler traffic was mostly hitting the edge to begin with, tuning TTL is wasted effort — which is exactly why we insist on measuring before acting.

Lever Two: CDN / WAF Rate Rules, Enforceable and Observable

The advantage of a rate rule is that it leaves a record: how many times it fired, against whom, in which window. That lets you run the rule as an experiment rather than setting it and closing your eyes.

The thing to watch is the allowlist. You don't want to throttle legitimate search engine crawling along with everything else, so those four official IP lists come back into play here — except this time you're using them to exclude, not to group. Which means the reliability of your list sync directly determines your collateral damage rate, and that 403 trap on the Anthropic list turns into a real loss at this layer. For where the settings live and how to work through them, follow along with our guide to Cloudflare CDN setup locations and steps.

How do you set the threshold? Our recommendation is to start slightly above the actual peak you measured, observe for one billing cycle, then tighten. Set it strict from day one and you won't be able to tell whether the traffic drop is the rule working or the rule causing damage.

Lever Three: Crawl-delay, Useful but Don't Count On It

Anthropic's documentation states that it supports Crawl-delay. But the nature of the directive is "declaring an expectation," not "enforcing" one — nothing guarantees the other side obeys, and there's no reporting mechanism to tell you whether it took effect.

More fundamentally: Crawl-delay isn't in the formal robots.txt standard at all. The IETF's RFC 9309, "Robots Exclusion Protocol" (Standards Track, September 2022) defines only three record types — user-agent, allow and disallow — and the string Crawl-delay appears nowhere in the document. Section 2.2.4 offers just one line on non-standard records of this kind: "Crawlers MAY interpret other records that are not part of the robots.txt protocol." MAY — they may interpret it, and they may ignore it entirely. So one vendor "supporting Crawl-delay" is goodwill it added itself, not a standards obligation. Who supports it, and how they interpret that number of seconds, is not something you're in a position to demand.

Its role is therefore clear: a supplement, not the main event. Setting it does no harm, but if your cost problem is genuinely serious, pinning your hopes on one advisory line is the same as not addressing it.

The Order of Operations

1. Tune cache TTL first — zero blocking risk, bank the savings that are available
2. Then set rate rules — set the threshold from the numbers you measured, pair it with an allowlist to limit collateral damage
3. Add Crawl-delay — cheap, treat it as a courtesy supplement
4. Go back and re-measure — run the same four steps again for origin GB and compare against your pre-change numbers

Step four is the one most often skipped, and it's the one that closes the loop. Was that saving really produced by your rules? Without re-measuring, you'll never be able to separate "the rule worked" from "that crawler just happened not to show up this month."

---

Which Setting to Choose Depends on the Plan You Bought

How far you can tune caching and rate rules is tied to which CDN you use and which plan you're on. CloudInsight resells AWS, GCP, Azure, Alibaba Cloud and Tencent Cloud, with unified billing, formal contracts, and Chinese-language technical support in the Taiwan time zone.

👉 Consult on enterprise plans now｜Join us on LINE for real-time consultation

---

Frequently Asked Questions

Q: How do I know whether my CDN traffic spike was caused by AI crawlers?

A: Start by grouping your logs by user-agent, then cross-check against the official IP lists. As measured on 2026-08-05, GPTBot publishes 21 prefixes and ChatGPT-User 289–290, so the comparison lets you attribute traffic to a specific crawler. Then look at cache status: if most of the spike consists of origin fetches rather than edge hits, the source is almost certainly not ordinary visitors.

Q: Can I calculate AI crawler traffic from User-Agent alone?

A: No. Anyone can forge a User-Agent string, so matching on it alone folds spoofed traffic into your numbers. Always cross-check against the official IP lists as well — OpenAI's gptbot.json and Anthropic's bots.json, for example. Note that this step yields grouping, not identity verification; real verification relies on mechanisms such as reverse DNS or cryptographic signatures.

Q: Does setting Crawl-delay actually do anything?

A: Anthropic's documentation states that it supports Crawl-delay, but the directive is advisory rather than enforceable, and there is no reporting mechanism to confirm whether the other side is honoring it. Among the three throttling levers it is the weakest, which makes it suitable as a supplement and unsuitable as your main measure. The lever that is both enforceable and observable is a CDN or WAF rate rule.

Q: If I block all AI crawlers, will my bandwidth bill definitely go down?

A: Not necessarily, and it may create new problems. Anthropic's documentation notes that blocking by IP may not achieve opt-out correctly or durably, because it prevents the crawler from reading your robots.txt. IP lists also drift on their own: during testing on 2026-08-05, ChatGPT-User's prefix count went from 290 to 289 within minutes, and a stale list causes collateral damage.

Q: How long do I need to retain logs?

A: Long enough to cover at least one complete billing cycle, otherwise the traffic you calculate won't line up with the period on your invoice. Seeing trends takes longer still, because crawling arrives in passes — keep only seven days and you may miss an entire pass. In practice, extend retention first to complete one full measurement, then tune it back based on log storage cost.

Conclusion: Measure First, Decide Second — Don't Block on a Hunch

Back to the question we opened with: how much bandwidth do AI crawlers actually cost you? From start to finish this article has answered exactly one thing — how to work out your own number — rather than telling you what somebody else's number is.

Three steps to close. First, turn on all five log fields (timestamp, user-agent, client IP, response bytes, cache status) and extend retention to cover at least one complete billing cycle. Second, run the four steps to get origin GB per bot, then match that against your bill to find the line item you're charged for. Third, tune cache TTL, then set rate rules, then add Crawl-delay — and go back and re-measure with the same method.

Once you have your own number, a lot of arguments dissolve on their own. Some people say AI crawlers are expensive, others say it makes no difference, and both are right — the only difference is that their site isn't your site. Your answer lives in your logs and on your invoice, nowhere else.

Caption: The order is measure, then adjust — the records on the left have to accumulate before the gauge in the middle can read anything, and only then is there a knob on the right worth turning.

---

🎯 Take Action Now

Turning AI crawler traffic from "it feels like a lot" into an identifiable line on your invoice is the fastest step you can take — and the one most often skipped.

👉 Consult now to get the plan that fits you best 👉 Join our official LINE account for real-time technical support

---

References

• IETF RFC 9309, "Robots Exclusion Protocol" (Standards Track, September 2022; the formal specification for robots.txt, retrieved 2026-08-05)
• OpenAI GPTBot IP list (gptbot.json) (measured 2026-08-05)
• OpenAI OAI-SearchBot IP list (searchbot.json) (measured 2026-08-05)
• OpenAI ChatGPT-User IP list (chatgpt-user.json) (measured 2026-08-05)
• Anthropic crawler IP list (bots.json) (measured 2026-08-05)
• Anthropic documentation: crawler behavior and how site owners can block it (measured 200 on 2026-08-05)
• Cloudflare Verified bots criteria documentation (page marked Last updated 2026-07-01)
• Cloudflare fake bot managed rules
• Cloudflare's announcement of new AI bot categories