6.2.2. Mount, LVM, and RAID Troubleshooting

💡 First Principle: A storage stack fails in layers, and the error you see is rarely reported by the layer that broke. A missing filesystem is usually a mount problem; a missing mount is often an LVM problem; an inactive volume group is often a disk problem. Diagnose downward — mount → LVM → RAID → physical device — because every layer above depends on the one below.

The storage dependency chain

physical disk → RAID array (md) → physical volume → volume group → logical volume → filesystem → mount point

Ask "what is the layer below telling me?" at each step. mount failing with "special device does not exist" is not a mount bug — it means the logical volume beneath it never activated.

# Mount failures
mount -a                         # Try mounting all fstab entries; shows errors
journalctl -b | grep "Failed to mount"   # systemd mount failures
systemctl status mnt-data.mount  # systemd mount unit status

# Common fstab issues
# - Wrong UUID (use blkid to verify)
# - Missing _netdev for NFS/CIFS (causes boot hang)
# - Wrong filesystem type
# - Missing mount point directory

# UUID verification
blkid /dev/sda1                  # Get actual UUID
grep UUID /etc/fstab             # Compare against fstab

# LVM not finding volumes
vgscan --mknodes                 # Scan for VGs and recreate device nodes
vgchange -ay                     # Activate all VGs
lvscan                           # Scan all LVs
pvscan                           # Scan all PVs

# RAID health
cat /proc/mdstat                 # RAID array status and rebuild progress
mdadm --detail /dev/md0          # Detailed status including failed devices
# States: active, degraded, rebuilding, failed
# degraded = running but missing a disk — replace ASAP
mdadm --manage /dev/md0 --remove /dev/sdb   # Remove failed disk
mdadm --manage /dev/md0 --add /dev/sdd      # Add replacement

RAID Rebuild Monitoring:

watch cat /proc/mdstat           # Live rebuild progress
# Shows: [====>............] recovery = 25.5% (...)
# Estimated time to completion also shown