6.4.1. Service Diagnostics and Dependency Chains

CPU and memory performance problems require distinguishing between different resource exhaustion patterns before reaching for a fix. The most dangerous trap is treating I/O wait as CPU load — the symptoms look identical in naive monitoring but the fixes are completely different.

Distinguishing CPU-bound from I/O-bound:

• CPU-bound: high %us or %sy in top/vmstat, low %wa
• I/O-bound: moderate %us but high %wa (wait) and elevated load average
• High load average with low CPU utilization almost always indicates I/O wait

Memory pressure indicators:

• free -h: focus on "available" column, not "free" — available includes reclaimable page cache
• Swapping (vmstat si/so columns non-zero) indicates real memory pressure
• OOM kills logged in dmesg or journalctl -k — the kernel chose a victim process
• oom_score_adj in /proc/PID/oom_score_adj controls kill priority (-1000 = never kill)

Per-CPU and multi-core visibility:

• top shows aggregate %CPU — values >100% indicate multi-threaded processes using multiple cores
• Press 1 in top to see per-CPU breakdown
• mpstat -P ALL 1 shows per-core utilization with I/O wait column

# Service status and recent logs
systemctl status nginx           # State + last 10 log lines
systemctl status nginx.service -l  # Full log lines (no truncation)
journalctl -u nginx --since "10 min ago"   # Last 10 minutes of nginx logs
journalctl -u nginx -f           # Follow live

# Failed service investigation
systemctl list-units --state=failed
systemctl status failed_service
journalctl -u failed_service -b  # Current boot logs
# Look for: ExecStart= failed, dependency failures, permission denied

# Service dependency issues
systemctl list-dependencies nginx     # What does nginx depend on?
systemctl show nginx | grep -E "After|Requires|Wants"  # Dependency config

# Process not starting — common causes
# 1. Port already in use
ss -tulnp | grep :80             # What's using port 80?
fuser 80/tcp                     # Which PID is using port 80

# 2. Permission denied
# Check that the service user can access required files
sudo -u nginx ls /var/www/html
sudo -u nginx cat /etc/nginx/nginx.conf

# 3. Config syntax error
nginx -t                         # Nginx
apachectl configtest             # Apache (apache2ctl on Debian/Ubuntu)
sshd -t                          # SSH

# 4. Missing dependency (library)
ldd /usr/bin/myapp               # List shared library dependencies
ldd /usr/bin/myapp | grep "not found"   # Missing libraries
ldconfig -p | grep libssl        # Is libssl in the cache?

# 5. SELinux denial
ausearch -m avc -ts recent       # Check for denials timed with service failure

Application Log Analysis:

# Real-time monitoring
tail -f /var/log/nginx/error.log
tail -f /var/log/nginx/access.log
multitail /var/log/nginx/error.log /var/log/app/app.log   # Multiple logs

# Pattern searching in logs
grep -c "ERROR" /var/log/app.log                  # Count errors
grep "ERROR" /var/log/app.log | awk '{print $1}' | sort | uniq -c  # Errors by date
awk '/ERROR/{count++} END{print count}' app.log

# Log parsing for performance analysis
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head  # Top URLs
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head  # Top IPs