5.5.1. System Performance Monitoring

💡 First Principle: System performance problems fall into four categories: CPU-bound (process needs more compute), memory-bound (not enough RAM — swap thrashing), I/O-bound (waiting for disk/network), and capacity (system simply doesn't have enough resources for the workload). The diagnostic goal is to identify which category applies before reaching for a solution.

# CPU and Load
uptime                           # Load averages: 1, 5, 15 minute
vmstat 1 5                       # Virtual memory stats: 1-second intervals, 5 samples
mpstat -P ALL 1                  # Per-CPU utilization
sar -u 1 10                      # CPU utilization (10 samples)

# Memory
free -h                          # Memory and swap usage
vmstat -s                        # Memory statistics summary
swapon --show                    # Swap devices and usage

# Disk I/O
iostat -xz 1                     # Extended disk I/O stats per device
iotop                            # I/O by process (like top for disk)
blktrace /dev/sda                # Low-level block device tracing
sar -d 1 10                      # Disk activity via sar

# Network
sar -n DEV 1 5                   # Network interface stats
nethogs                          # Network usage by process (like top for network)
iftop                            # Network traffic by connection

# Comprehensive tools
sar                              # System Activity Reporter — collects everything
atop                             # Advanced: CPU, memory, disk, network per process
dstat                            # Combined stats (replaces vmstat/iostat/netstat)

Understanding Load Average:

Load average represents the average number of processes in the run queue (running or waiting for CPU) over the last 1, 5, and 15 minutes. On a 4-core system:

• Load average of 4.0 = 100% utilized (one process per core)
• Load average of 8.0 = 200% — system is overloaded
• Load average > CPU count = performance problem
• Load average < CPU count = capacity available

Note: I/O-waiting processes also count toward load average in Linux, so high load can mean disk/network saturation, not just CPU saturation.