The Record Layer Is Part of Your Recovery Plan: Drilling DNS Before It Drills You

Most UK digital teams have a recovery plan for the application tier. They know which database replica to promote, which CDN rule to bypass, and who declares an incident. The record layer often sits outside that plan. DNS is treated as background configuration rather than a dependency with failure modes, change windows and trust consequences. That is a gap worth closing before a routine registrar change or a DNSSEC rollover becomes a customer-facing outage.

Why the record layer escapes rehearsal

DNS changes are infrequent, often owned by a small group, and usually mediated by a registrar or third-party provider. That makes them easy to exclude from game days. The result is a dependency that is invisible until it breaks, at which point support queues fill with vague connectivity reports and the organisation discovers it has no tested path from registrar to resolver.

A practical drill does not need to simulate a full outage. It needs to prove that the team can change a record, validate it across resolver populations, roll it back and explain what happened. The aim is engineering-minded reassurance: fewer assumptions, more evidence.

Set scope and pass marks first

Choose one production domain or a low-risk subdomain that carries realistic traffic. Define the scenarios in advance: a registrar lock check, a TTL-led record change, a DNSSEC key rollover rehearsal, and a secondary name server failover. Agree pass marks before the drill: record changes visible within the planned window, no unexplained SERVFAIL responses, rollback completed, and evidence captured for the incident record.

Include the people who can actually unlock the domain, change the zone and update the status page. A drill that only involves the network team misses the escalation path that matters at 02:00.

Registrar controls: lock, roles and recovery

Start with the registrar. Confirm that clientTransferProhibited is set, that registry lock is enabled where the registrar offers it, and that multifactor authentication is enforced for every account that can alter nameservers or DS records. Check recovery email addresses and billing contacts; an expired card can suspend a domain just as surely as a bad A record.

Do not unlock a production domain simply to prove you can. Use a staging domain with the same registrar controls, or run a tabletop exercise with the registrar's support team. Record who can authorise an unlock, how long it takes, and what identity evidence is required. If the answer is one person's mobile phone, the dependency is under-rehearsed.

DNSSEC rollover without drama

DNSSEC turns a quiet misconfiguration into hard failure for validating resolvers. Rehearse the sequence on a low-risk zone: pre-publish the new DNSKEY, update the DS record at the registry, allow the old key to remain valid through the propagation window, then retire it. Monitor with a validating resolver and with a resolver that does not validate, so you can distinguish a DNSSEC fault from a general resolution problem.

For .uk domains, remember that the DS record lives at the registry while the DNSKEY lives at the DNS provider. The handshake between those two organisations is part of your recovery plan. A rollover drill should include the registrar's interface, the provider's key management screen and the person who can confirm the DS digest.

TTLs are a recovery lever, not a constant

TTL strategy is often inherited rather than designed. A low TTL before a planned change gives you a shorter rollback window; a permanently low TTL adds query load and does not guarantee instant propagation. Inspect TTLs for apex, www, API and CNAME records, and check the SOA minimum for negative caching. Rehearse a change from a short TTL to a longer one after the window closes.

Propagation is not a single event. It is a distribution across recursive resolvers, corporate networks, mobile operators and public resolvers. A propagation check should sample those populations, not just the resolver on the engineer's laptop.

Secondary providers and third-party name server risk

A secondary DNS provider is useful only if it is genuinely independent. Two providers that share an upstream, an account owner or a control plane may share the same failure. During the drill, fail traffic to the secondary in a controlled way, confirm that zone transfers are current, and check that the secondary returns the same DNSSEC signatures and TTLs.

Third-party name servers bring their own risk. A SaaS provider's nameservers may be reliable, but they are still a dependency outside your change control. Record the provider, the contract owner, the escalation route and the last successful zone transfer. If the provider cannot give you a tested failover path, treat that as a known risk rather than an assumed capability.

A .uk-specific view of the record layer

Nominet operates the .uk registry, and the registrar is your commercial and operational interface to it. That distinction matters during a drill: the registry may be healthy while the registrar's control panel is unavailable, or the reverse. Keep registrar and registry escalation paths separate, and test the registrar's out-of-hours process before you need it.

UK resolver populations are varied. Corporate networks may run their own resolvers with aggressive caching, mobile operators may behave differently, and public resolvers may validate DNSSEC more strictly. Sample them deliberately. Tools such as dig +trace, DNSViz and distributed probe networks give more useful evidence than a single cached lookup.

A quarterly runbook that produces evidence

Before the drill, inventory every domain, record type, TTL, provider and owner. Freeze unrelated changes, brief the support desk, and agree the rollback trigger. During the drill, capture timestamps, resolver responses, screenshots and the name of the person who approved each step. Afterward, record time to detect, time to change, validation coverage and the number of resolver populations checked.

For Site Confidence, the measure of a good DNS drill is not a dramatic failover. It is a calm, documented change that leaves the organisation with stronger evidence and fewer unknowns. The record layer will never be glamorous, but it can be rehearsed, observed and trusted.