The Gatekeeper's Ledger: Auditing Bot Defences as a Customer-Journey Control

Bot management is usually procured as a barrier. The pitch is straightforward: make automated abuse expensive, keep credential stuffing and scraping away from the origin, and let genuine customers through. In practice, the challenge screen is not a wall at the edge of the estate. It is a control in the middle of a customer journey, and every false positive becomes a service ticket, an abandoned basket or a failed recovery attempt.

UK digital teams therefore need a different testing habit. Instead of asking only whether the vendor blocks bad traffic, ask what the bot defence does to the customers it is supposed to protect. That means keeping a ledger: which journeys are challenged, how often legitimate users are stopped, how long the pause lasts, who is excluded by the interface, and how much support effort the control generates.

Begin with journey scripts, not vendor demos

A bot defence cannot be assessed in isolation. Define a small set of realistic scripts and run them repeatedly against a production-like environment. For most UK organisations, four scripts cover the highest-risk surfaces:

• A returning customer logging in with a password manager on a familiar device.
• A guest checkout on a mobile connection, using a saved card and a UK delivery address.
• An account recovery flow that starts with an email link and ends with a password reset.
• A low-frequency but high-value action, such as changing a delivery address or downloading a statement.

Add identity states rather than just devices: new device, corporate VPN, mobile roaming, older browser, assisted digital user, and a synthetic account with a known-good history. The point is not to simulate attackers. It is to simulate customers whose behaviour looks unusual for benign reasons.

Accessibility is a functional requirement, not a checkbox

WCAG 2.2 AA changed the conversation for authentication. Success criterion 3.3.8, Accessible Authentication (Minimum), requires that authentication does not rely on a cognitive function test unless an alternative or assistance is available. Image-grid puzzles, distorted text and audio-only fallbacks often fail that test in practice. So do challenge flows that block paste into password fields, interrupt password manager autofill, or impose short time limits without an extension mechanism.

Test with the tools customers actually use: keyboard only, NVDA and JAWS on Windows, VoiceOver on macOS and iOS, TalkBack on Android, voice control such as Dragon, and switch access. Record the exact point of failure. “CAPTCHA is inaccessible” is not a finding; “the audio alternative requires a second challenge that cannot be reached by keyboard, and the visual grid has no text alternative” is.

For UK public sector bodies, the Public Sector Bodies (Websites and Mobile Applications) Accessibility Regulations 2018 make this a compliance matter. For private-sector services, the Equality Act 2010 and the consumer duty expectations in regulated markets make it a conduct risk as well as a conversion problem. A gate that excludes disabled customers is not a neutral security control.

Keep a false-positive ledger

False positives are the cost of a challenge that should not have been shown. Measure them in three stages: challenge presented, challenge passed first time, and challenge abandoned. Segment by device class, network type, browser, assistive technology, account age and geography. Use canary accounts that should always pass, and run them on a schedule so that a vendor rule change does not silently tighten the gate.

Set starter thresholds against your risk appetite. A reasonable opening position is that no more than 0.5% of authenticated returning customers should see a challenge in a normal week, and no more than 0.1% should fail a challenge they should have passed. Those numbers will move with threat levels, but the ledger makes the movement visible. Without it, teams negotiate with vendors using anecdotes.

Measure latency as part of the journey budget

A challenge adds render time, interaction time and verification time, often across third-party infrastructure. Measure p50, p75 and p95 for each stage, and include the challenge in the end-to-end journey budget rather than treating it as an edge concern. On UK mobile networks, a p95 challenge render of more than 2.5 seconds before the customer can interact is a warning sign, particularly on checkout and recovery journeys where intent is fragile.

Keep the measurement honest. Disable browser caching in the same way a first-time customer would experience the flow, test on throttled connections, and record whether the challenge blocks the main thread or shifts layout. A security control that degrades Core Web Vitals is still a performance regression, even if the vendor dashboard reports a successful verification.

Follow the support load

The service desk is where false positives become visible. Tag contacts by journey and symptom: “could not complete security check”, “verification code not received”, “login loop”, “payment page refreshed”. Then calculate contact volume per 10,000 sessions, average handling time, repeat contact rate within 72 hours, and the proportion of contacts that end in a manual override.

Compare periods before and after a vendor rule change, a new challenge type or a peak trading event. An interrupted time-series view is usually enough to show whether a security improvement has moved cost into the contact centre. That matters for UK retailers in the run-up to Black Friday and Christmas, when a small increase in challenge abandonment can translate into a large queue the following morning.

Step-up checks need a recoverable path

Device fingerprinting and risk scoring are less visible than a CAPTCHA, but they can create harder dead ends. A customer who is silently challenged and then asked for a one-time code may have no fallback if the phone number is old or the email is inaccessible. Test the recovery path under load, not just the happy path.

Fingerprinting also raises data protection questions. Device and behavioural signals can constitute personal data, so document the lawful basis, retention period and vendor data flows, and keep the evidence in the same ledger as the security test results. The ICO’s accountability expectations apply to fraud prevention as much as to marketing.

A scorecard that earns trust

Bring the evidence together in a quarterly scorecard: journey, challenge type, accessibility pass or fail, false-positive rate, p95 latency, support contacts per 10,000 sessions, fallback quality and named owner. Review it after any material change and before peak trading. Use it in vendor reviews. A bot defence that blocks customers is a reliability incident with a security label, and it deserves the same engineering discipline as any other customer-journey control.