Backup and disaster recovery you have actually tested

Assumptions, and when this guide does not apply

This guide is for teams running their own servers, VPS, bare metal or cloud instances, who are responsible for at least part of their backups. It assumes you can list your systems, their data sizes and who depends on them.

It is not enough on its own when:

• A regulation sets retention or recovery terms (financial records, health data, government contracts). Those rules override the planning defaults here; CISA notes that immutable storage does not meet compliance criteria for certain regulations, so check before locking data.
• You need zero data loss across sites. That is synchronous replication and application design, not backup. Backups protect against loss and corruption; replication protects availability.
• The data lives in a SaaS product. You may need a separate export or backup service; the hosting design here does not cover it.

Who does what (backups, restore testing, key custody) must be written down. See Shared responsibility in hosting and Security and reliability.

RPO, RTO and MTD: three numbers that drive the design

NIST SP 800-34 Rev. 1, the US federal contingency planning guide, defines the terms most teams use:

• Recovery point objective (RPO): "the point in time to which data must be recovered after an outage." An RPO of 1 hour means you accept losing up to 1 hour of changes. It sets how often you take backups.
• Recovery time objective (RTO): "the overall length of time an information system's components can be in the recovery phase before negatively impacting the organization's mission or mission/business processes." It sets how fast you must restore.
• Maximum tolerable downtime (MTD): "the amount of time mission/business process can be disrupted without causing significant harm to the organization's mission." The RTO for a system must fit inside the MTD of the business process it supports, with room for detection and decision time.

Set the numbers per system, not per company. An order database and a marketing site have very different costs of downtime, and paying for a 15-minute RPO everywhere is waste.

Replicas are not backups. A DROP TABLE or ransomware encryption reaches the replica in seconds. Keep a replica for RTO and a backup for RPO.

3-2-1 and 3-2-1-1-0: what each digit defends against

CISA's Data Backup Options paper states the 3-2-1 rule: keep 3 copies of any important file (1 primary and 2 backups), on 2 different media types, with 1 copy offsite. The rule was designed around hardware failure and site loss. Modern attacks target the backups themselves, so many teams extend it to 3-2-1-1-0, an industry convention rather than a standard.

The offsite copy should not share credentials with production. If the same admin account can delete the server and its offsite backups, you have one copy in practice.

Immutability and object lock

Object lock stores backups write-once-read-many (WORM): once written, an object version cannot be changed or deleted until its retention date passes. The S3 API's Object Lock, implemented by many S3-compatible object stores, is the common form. Its behavior, from the AWS reference documentation:

• Versioning is required. Lock protects a specific object version. A simple delete adds a delete marker on top; the locked version stays restorable.
• Governance mode: users with a special bypass permission can still shorten retention or delete. Good for testing settings, weaker against a stolen admin credential.
• Compliance mode: no user, including the account root user, can delete the version or shorten retention until it expires. Strongest protection, but a wrong retention setting cannot be undone, and you pay for storage until it expires.
• Legal hold blocks deletion with no end date until someone with permission removes it.

Set retention longer than the time an intruder might go unnoticed before acting. If your logs would only show an intrusion after 21 days, 14 days of locked retention can leave you with only infected restore points.

Ransomware: plan for an attacker who finds your backups

CISA's #StopRansomware Guide (September 2023) recommends maintaining offline, encrypted backups of critical data and regularly testing their availability and integrity in a disaster recovery scenario, because many ransomware variants try to find and delete or encrypt accessible backups. It also advises keeping "golden images" of critical systems up to date. NIST's Cybersecurity Framework 2.0 lists the same outcome as PR.DS-11: backups of data are created, protected, maintained and tested.

What that means in practice:

• Separate identity. Backup administration uses different accounts, with MFA, outside the main directory domain. The production server's credentials can write new backups but cannot delete old ones.
• Encryption keys stored apart. Keep backup encryption keys and the recovery runbook somewhere that survives the loss of production, and test that the people on call can reach them.
• Golden images and configuration as code. Rebuilding a clean server from a known image is faster and safer than restoring a whole compromised disk.
• Restore into isolation. Bring restored systems up on a separate network, scan them, rotate credentials, and only then reconnect.
• Know your clean restore point. Keep several restore points across days and weeks so you can step back past the intrusion date.

How long a restore really takes: the arithmetic

Transfer time is usually the largest single block of an RTO, and the most often underestimated. Do the sum before you sign up to a recovery target.

The link is not the only limit. Restore throughput is the slowest of: the network path, the backup store's read rate, decompression and decryption CPU, and the target disk's write rate. Millions of small files add per-file overhead: 10 million files at 2 ms each, processed one at a time, add 20,000 seconds (5.6 hours). Parallel streams and image-level restores reduce this.

Restore testing cadence

Untested backups fail in predictable ways: missing files excluded by a pattern, expired encryption keys, a database dump that was never consistent, a restore that takes three times longer than planned. These cadences are planning defaults; tighten them for systems with short RTOs.

A restore-drill runbook

1. Set scope and targets. Pick one system. Write down its RPO and RTO, the restore point you will use, and what "working" means (pages load, logins work, last order visible).
2. Start the clock. Record the start time. Every later step gets a timestamp.
3. Retrieve access as if production were gone. Get backup credentials and encryption keys from where they are stored for emergencies, not from a production server. If this step fails, the drill has already found its most important gap.
4. Provision an isolated target. A new server or VM on a network with no route to production, built from your golden image or configuration code.
5. Restore data. Record bytes restored and elapsed time; calculate achieved throughput and compare it with the figure in your RTO budget.
6. Recover the application. Replay database logs to the target point, start services, run migrations if needed.
7. Verify. Checksums, database consistency checks, row counts against a known reference, application smoke tests, and a person checking the business result.
8. Measure achieved RPO and RTO. RPO: time of the last transaction present versus the chosen incident time. RTO: start to verified service.
9. Tear down and record. Destroy the isolated environment and its credentials. Write a short report: targets, achieved numbers, failures, fixes, owner and date for each fix.
10. Update the runbook. Change the written steps to match what actually worked, and schedule the next drill.

Backup and recovery checklist

• RPO, RTO and MTD written down for each system, with a business owner who agreed them.
• Backup mechanism per system matches its RPO (database log archiving where minutes matter).
• Three copies, two platforms, one offsite under separate credentials.
• One immutable or offline copy with retention longer than your likely detection time.
• Backup deletion requires a different identity, with MFA, from production administration.
• Encryption keys and runbook available without production.
• Restore time calculated from data size and measured throughput, and inside the RTO.
• Weekly automated restore test and a quarterly timed restore drill on the calendar.
• Responsibility for backups, restore tests and key custody written down between you and your provider.
• Numbers checked in the recovery feasibility tool.

Common mistakes

• Calling a replica a backup. It copies deletions and encryption as fast as it copies good data.
• Backups in the same account as production. One stolen admin credential removes both.
• Retention shorter than detection time. Every restore point may already contain the intruder.
• RTO set without the transfer sum. 2 TB at 100 Mbit/s takes more than 2.5 days at 70% efficiency.
• Testing that the job ran, not that data restores. A green backup job does not prove a usable restore.
• Keys stored only on the server being protected. Encrypted backups without the key are not backups.
• Compliance-mode lock set before testing retention. It cannot be shortened, and storage is billed until expiry.

Related