5.5.2. Observability Stacks and Alerting

Modern Observability: Three Pillars:

Prometheus + Grafana:

# Node Exporter — exposes Linux metrics for Prometheus
# Installed on every monitored node; listens on port 9100
systemctl enable --now node_exporter

# Prometheus scrapes metrics from exporters
# grafana visualizes metrics from Prometheus
# alertmanager handles alerting rules

# Check metrics endpoint
curl http://localhost:9100/metrics | grep node_cpu

Log Aggregation (ELK Stack):

• Elasticsearch — stores and indexes log data
• Logstash — processes and transforms log data
• Kibana — visualizes logs from Elasticsearch
• Filebeat — lightweight log shipper (installed on each node)

Alerting Concepts:

# Nagios/Icinga — traditional monitoring
# Alert conditions: check_disk, check_load, check_http
check_disk -w 20% -c 10% -p /   # Warn at 20% free, critical at 10% (bare numbers would mean MB, not %)

# systemd watchdog — restart failing services automatically
# In unit file:
# WatchdogSec=30s        — systemd kills/restarts if no keepalive for 30s
# Restart=on-failure     — restart if service exits non-zero

# SNMP — Simple Network Management Protocol
# For monitoring network devices and legacy systems
snmpwalk -v2c -c public 192.168.1.1 .1.3.6.1.2.1.1.1   # Get sysDescr

⚠️ Exam Trap: A high load average alone doesn't identify the cause — you need to distinguish CPU saturation (%usr and %sys in mpstat) from I/O wait (%iowait). vmstat column wa shows the percentage of time CPUs are idle waiting for I/O. High wa with low us/sy means I/O is the bottleneck, not CPU — adding more CPUs won't help.

Reflection Question: uptime shows a load average of 24 on a 4-core server. top shows %iowait at 85% and CPU %us at 5%. What is the bottleneck and what diagnostic tools would you use next?