Uninvited Guests: Load-Testing for AI Crawlers and Agentic Shoppers Before They Test You

At 07:40 on a damp Wednesday in early November, origin CPU on a UK retailer's product listing service climbs from 22% to 61% in nine minutes. There is no deployment, no marketing email and no television appearance. The traffic is well-formed, persistent, and entirely uninterested in buying anything.

This is now a routine morning. AI assistants summarising ranges, price aggregators refreshing catalogues, SEO tooling sampling faceted pages, agentic browsers navigating on behalf of a user who may never arrive. They consume the same origin capacity, the same database connections and the same search cluster as the customers you built the estate for. Yet most performance test plans still simulate two populations: humans and search engines. The third population is invisible in the model and very visible in the graphs.

Start With a Census, Not a Blocklist

Blocking before measuring is the most common and most expensive mistake. Spend a fortnight instrumenting instead.

Pull thirty days of edge logs and derive four things: crawler share of total requests, crawler share of uncached origin requests, the top twenty URL templates being crawled, and cache hit ratio per template. The second figure is the one that matters. A bot fetching cached HTML is close to free; a bot fetching an uncached, session-personalised listing page is a full stack traversal with your name on it.

Treat the user-agent string as a claim, not a fact. Verify by IP ownership, reverse DNS and ASN grouping, and cross-check against the published bot directories that the major platforms maintain. Group what remains into four families: verified AI agents, verified search crawlers, unverified scrapers, and your own monitoring. Each will need a different policy.

Cache Is Still the Cheapest Lever You Own

Most crawler pressure disappears with mundane cache hygiene. Normalise query strings so tracking parameters do not create new cache keys. Serve anonymous GETs from the edge with sensible s-maxage and stale-while-revalidate. Keep price and stock in separately keyed fragments so a single volatile element does not invalidate an entire listing page. Above all, stop letting a session cookie fragment the cache for visitors who have not signed in.

Measure before and after. In our experience a normalisation pass and a fragment split routinely cut origin requests attributable to crawlers by half, without a single block rule and without any change to what a customer sees.

Controlled Experiments Beat Policy Guesswork

Once you can count, test. Run the matrix against a production-like replica where possible, or in production behind a canary with a rollback switch at the edge. Test one bot family at a time and avoid the fortnight before Cyber Weekend; nobody wants to discover a misconfiguration mid-peak.

The four configurations worth measuring:

1. Identify only. No enforcement. Establishes the baseline and proves your tagging is accurate.
2. Throttle. Rate-limit verified AI agents to a request-per-second ceiling and watch origin CPU and human p75 latency.
3. Path restriction. Deny or challenge expensive templates — faceted search, add-to-basket, account endpoints — while leaving catalogue and product pages open.
4. Challenge unverified. Apply a managed challenge or block to traffic that fails verification and is not a known partner feed.

For each, record origin CPU, error rate, human latency percentiles and — critically — whatever downstream discovery signal you can observe. A test that improves CPU while silently removing your products from an AI answer surface is not a win; it is a trade nobody authorised.

The Discovery Ledger

Blocking has a cost, and it tends to arrive weeks later in a channel nobody owns. Keep a short ledger of what you blocked, who approved it, the rollback date and the discovery metrics you agreed to watch: referrals from AI assistants, catalogue freshness in comparison services, crawl frequency of priority pages, affiliate and merchant feed health.

There is also a consistency point that touches trust. Declaring one policy in robots.txt while quietly enforcing something stricter at the edge is a contradiction an external party can spot, and increasingly will. Decide the policy, state it, enforce it in the same shape, and publish a contact route for partners who believe they have been wrongly caught.

A Five-Stage Plan for UK Teams

Stage one: baseline. Instrument for two weeks. Change nothing.

Stage two: taxonomy. Classify every non-human family and assign an owner.

Stage three: cache remediation. Take the cheap capacity back first and re-measure.

Stage four: controlled enforcement. One family, one rule, one rollback, reviewed within a week.

Stage five: publish and review. Update robots.txt, terms and bot management rules together, then revisit quarterly alongside your peak readiness cycle.

Reporting Upwards

A single board slide does the job: crawler share of origin requests, the capacity that represents in cost or headroom, the proportion now blocked or throttled, the observed discovery impact, and the decision you need. Framed that way, the conversation stops being about bots and becomes about capacity discipline and earned trust — which is where it belongs.

The goal is not exclusion. It is a deliberate budget for traffic you did not invite, set by measurement rather than by irritation.