6.4.2. Backup Verification and Disaster Recovery

💡 First Principle: An untested backup is not a backup — it's a backup attempt. The only way to know a backup works is to restore from it in a controlled environment. Backup verification should be scheduled, automated, and logged. The restore test doesn't need to be a full production restore — restoring a single file or a sample of files proves the backup is readable and complete.

Backup Verification:

# Verify tar archive integrity (no extraction)
tar -tzf backup.tar.gz > /dev/null && echo "Archive OK" || echo "CORRUPT"
tar -tvzf backup.tar.gz | wc -l  # Count files in archive

# Verify checksums
sha256sum -c backup.sha256       # Verify all files against checksum file
md5sum -c backup.md5

# Test restoration
mkdir -p /tmp/restore_test && tar -xzf backup.tar.gz -C /tmp/restore_test/   # Restore to temp location (-C needs an existing dir)
diff -r /original/path/ /tmp/restore_test/     # Compare restored vs original

# rsync-based backup verification
rsync --dry-run -av --delete /source/ /backup/ | head -20   # Preview differences

# Database backup verification
mysqldump mydb | mysql -u root test_restore_db  # Restore to test DB
pg_restore -d test_db backup.dump               # PostgreSQL restore test

Disaster Recovery Concepts:

MTTR and MTBF:

• MTTR (Mean Time To Recover): Average time to restore service after a failure. Minimize with runbooks, automation, and monitoring.
• MTBF (Mean Time Between Failures): Average time between failures. Increase with redundancy, quality hardware, and preventive maintenance.

System Rescue:

# Boot into rescue mode (systemd)
# Append to kernel cmdline in GRUB: systemd.unit=rescue.target

# chroot into installed system from rescue media
# Mount root partition first
mount /dev/sda1 /mnt
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt                      # Now running in the installed system's context
# Now you can: reinstall GRUB, fix /etc/fstab, reset root password

# Reset root password (while chrooted or in single-user mode)
passwd root                      # Set new root password

Root Password Recovery (standard procedure):

1. Reboot → GRUB menu → Press e to edit
2. Find the linux line → append rd.break (RHEL) or init=/bin/bash (Debian)
3. Boot into emergency shell
4. Remount root read-write: mount -o remount,rw /sysroot (rd.break) or mount -o remount,rw / (init=/bin/bash)
5. With rd.break only: chroot /sysroot (with init=/bin/bash the root filesystem is already / — no chroot)
6. Reset password: passwd root
7. On SELinux systems: touch /.autorelabel (forces relabeling on next boot)
8. exit; exit → system reboots with new password

⚠️ Exam Trap: After resetting the root password on an SELinux-enabled system without running touch /.autorelabel, SELinux will deny login — the /etc/shadow file will have the wrong SELinux context after the password change in the emergency environment. Always create .autorelabel before rebooting from recovery.

Reflection Question: You need to recover a server that won't boot past the kernel panic "VFS: Unable to mount root fs." The disk appears healthy and SMART shows no errors. What are the three most likely causes of this specific kernel panic and what is the first recovery step for each?