Skip to content
Santekno.com | Tech Tutorials and Trends
EN
📖 0%
24 Aug 2025 · 6 min read ·Article 55 / 125
Go

55 Monitoring and Observability with Prometheus

IH
Ihsan Arif
Writer at Santekno · Backend Engineer

55 Monitoring and Observability with Prometheus: A Complete Guide for Engineers

Observability has become a buzzword in the world of DevOps and Site Reliability Engineering (SRE). As modern application architectures grow ever more complex—from microservices and Kubernetes to cloud-native—conventional monitoring is no longer enough. Observability is not just about collecting metrics; it is also about understanding your systems through data, visualization, and actionable alerts. One of the most popular open-source tools for this purpose is Prometheus.

In this article, I will share 55 monitoring and observability practices you can carry out with Prometheus, complete with code examples, simulations, tables, and flow diagrams. The goal is that you, as an engineer, don’t just “install a monitoring tool,” but truly understand and control your systems proactively.

What Is Prometheus?

Prometheus is an open-source monitoring and alerting system originally developed by SoundCloud and now part of the CNCF. Prometheus focuses on storing time series data (metrics) collected via HTTP pull (scraping)—a great fit for modern applications that scale dynamically.

At a high level, the Prometheus architecture looks like this:

MERMAID
flowchart LR
    subgraph Data Sources
        A1[App 1]
        A2[App 2]
        A3[App N]
    end
    A1 --> Scrape[Prometheus Scraper]
    A2 --> Scrape
    A3 --> Scrape
    Scrape --> TSDB[(Time Series DB)]
    TSDB --> UI[Prometheus UI]
    TSDB --> Alert[Alertmanager]
    UI --> Dev[Engineer/SRE]
    Alert --> Pagerduty[(PagerDuty/Email/Slack)]

Why Is Observability Important?

Without adequate observability:

  • Incidents are easily missed.
  • Root causes are hard to find.
  • Reliability is difficult to control at high scale.

With Prometheus-based observability:

  • Your system can proactively detect problems.
  • You get rich, query-able metric data.
  • You gain integrations for Alerts and Dashboards (Grafana).

55 Monitoring and Observability Metrics with Prometheus

The following collection of metrics can be grouped into three categories: Infrastructure, Application, and Business.

CategoryMetric Type (Example)Brief Description
InfrastructureCPU usage, Memory, Disk IO, NetworkServer/VM/container/Kubernetes Node basics
ApplicationHTTP Req/sec, Latency, Error Rate, QueueApplication health and performance
BusinessOrder Count, Transaction Success, Cart AddImportant business insights to monitor

Here is a list of popular monitoring and observability metrics you can implement:

  1. CPU Usage per Node
  2. CPU Usage Container (Pod)
  3. Memory Usage per Node
  4. Memory Usage Container
  5. Disk Usage
  6. Disk IOPS
  7. Network Receive/Transmit (Bytes/sec)
  8. Network Packet Error Rate
  9. Node Uptime
  10. Process Count (Zombie, Running)
  11. Heap Memory Consumption (Java Applications)
  12. Garbage Collection Time
  13. Thread Count (Application)
  14. HTTP Request Rate
  15. HTTP Error Rate (4xx, 5xx)
  16. HTTP Average Latency
  17. HTTP P95/P99 Latency
  18. Database Query per Second
  19. Database Query Errors
  20. Connection Pool Size
  21. DB Locks
  22. Redis Cache Hit/Miss Ratio
  23. Message Queue Size
  24. Queue Consumption Rate
  25. Queue Failure Count
  26. External API Call Rate
  27. External API Call Latency
  28. Job Scheduler Delay
  29. Kube Pod Count (Running, Pending, Failed)
  30. Kube Deployment Replicas Missed
  31. Kubernetes Node NotReady Count
  32. Container Restart Rate
  33. Volume Attach Delay (Storage)
  34. Cert Expiry Date
  35. SSL Handshake Error
  36. Ingress Traffic per Path
  37. TLS Version Usage
  38. CPU Throttling Events
  39. OutOfMemory Killed Count
  40. Disk Space Remaining
  41. Service Discovery Changes
  42. DNS Failures
  43. Config Reload Events
  44. Environment Variable Missed
  45. Feature Flag State
  46. Deployment Event Timestamps
  47. Rolling Update Progress
  48. Order Created per Second
  49. Transaction Success Rate
  50. Payment Failure Rate
  51. Users Logged In
  52. Active Sessions
  53. Shopping Cart Add/Remove Event
  54. Revenue per Minute
  55. Business SLA Compliance (e.g., <500ms resp)

Code Example: Implementing Custom Metrics in an Application

Prometheus Client Library for Python

Suppose you build a Python Flask web application. You can expose the following custom metrics:

python
 1from prometheus_client import start_http_server, Summary, Counter
 2from flask import Flask
 3
 4# Metric: Tracking HTTP request latency and error
 5REQUEST_LATENCY = Summary('http_request_latency_seconds', 'HTTP Request latency')
 6ERROR_COUNTER = Counter('http_error_count', 'Number of HTTP failed responses', ['code'])
 7
 8app = Flask(__name__)
 9
10@app.route("/api")
11@REQUEST_LATENCY.time()  # Automatically track latency
12def api():
13    # Simulate an error
14    if random.random() < 0.1:
15        ERROR_COUNTER.labels(code='500').inc()
16        return "Internal Error", 500
17    return "OK"
18
19if __name__ == "__main__":
20    start_http_server(8000)  # expose metrics at /metrics
21    app.run(host="0.0.0.0", port=8080)

The application will expose a /metrics endpoint that Prometheus can scrape.

Example Prometheus Scrape Config

Add the service to prometheus.yml:

yaml
1scrape_configs:
2  - job_name: 'webapp'
3    static_configs:
4      - targets: ['webapp:8000']

Simulating a Grafana Dashboard

Visualization with Grafana is essential for quick insights. You can build PromQL queries such as:

  • Error Rate:
    promql
    1sum(rate(http_error_count[5m]))
  • Average HTTP Latency:
    promql
    1avg(rate(http_request_latency_seconds_sum[5m])) / avg(rate(http_request_latency_seconds_count[5m]))

Alarms/Alerts with Alertmanager

Here is an example alert rule that fires when the error rate exceeds 0.5/s for 5 minutes:

yaml
 1groups:
 2- name: webapp-alerts
 3  rules:
 4  - alert: HighErrorRate
 5    expr: sum(rate(http_error_count[5m])) > 0.5
 6    for: 5m
 7    labels:
 8      severity: critical
 9    annotations:
10      summary: "High Error Rate Detected"
11      description: "The average error rate exceeded 0.5/s for 5 minutes"

Mini Case Study: Debugging a Slow Checkout

Suppose you discover that the checkout SLA of an e-commerce application frequently fails. Here is a monitoring and observability approach you can take:

MERMAID
flowchart TD
    A[Customer Checkout Request] --> B[App HTTP Req]
    B --> C[Database Query]
    C --> D[Redis Fetch]
    D --> E[External Payment API]
    B -.metrics.-> F[Prometheus Collects: latency, errors, queue]
    E -.metrics.-> F

    F --> G[Grafana Dashboard]
    G --> H[Alertmanager (High Latency Detected)]
    H --> I[SRE Team Triage]

The results of observing the metrics:

  • Checkout P99 latency rose significantly,
  • Payment API errors increased,
  • Redis cache misses spiked.

The root cause can be analyzed quickly—for example, Redis was down, so the fallback to the database was slow.

Conclusion

Modern monitoring and observability are no longer just about “knowing the server is down.” Prometheus and its ecosystem can unify infrastructure, application, and business metrics in a single centralized observability dashboard. At the very least, the 55 key metrics above are worth considering for large-scale production. With Prometheus, you can be more proactive, data-driven, and ready to respond to service issues.

Danger
“You can’t improve what you can’t measure.”
– Peter Drucker

If you want to dive deeper into observability, make sure every engineer on your team can read, add to, and experiment with metrics—because in the end, observability is a team capability, not just a tool.

Share your observability experiences in the comments. Which tools and metrics do you think are most crucial in a production environment?

Related Articles

💬 Comments