70. Case Study: Health Check Monitoring via Prometheus

Author: Bagus Pratama – Senior DevOps Engineer

One of the common challenges when dealing with distributed systems is making sure every service stays healthy at all times. Monitoring becomes mandatory, and the “health check” is a crucial aspect so we can quickly find out if a service stops functioning. In this article, I’ll share a case study of implementing health check monitoring using Prometheus—from the introduction and design to implementation and incident simulation, complete with code examples and diagrams.

Why Prometheus?

Prometheus has become the de-facto standard for monitoring and alerting at many technology companies. It has a large ecosystem, integrates with many services, and is a great fit for time-series monitoring—including monitoring health check endpoints via custom metrics or standard metrics.

Case Study: Monitoring the “Order API Service”

Imagine you have a service called order-api that an e-commerce application uses to handle the order process. This service runs on a Kubernetes cluster. To make sure the service stays healthy, every minute Prometheus scrapes the /healthz endpoint of order-api.

We want to gain the following visibility:

• UP/DOWN status of the health check (binary).
• Health check response time.
• An alert if the health check fails more than once within 5 consecutive minutes.

High-Level Design

Let’s visualize the flow of this monitoring system with a diagram:

flowchart TD
    subgraph Service Layer
        A[order-api /healthz endpoint]
    end
    subgraph Monitoring Layer
        B[Prometheus Scraper]
        C[Grafana Dashboard]
        D[Alertmanager]
    end
    A -- expose metrics --> B
    B -- store & query --> C
    B -- trigger alert --> D

flowchart TD
    subgraph Service Layer
        A[order-api /healthz endpoint]
    end
    subgraph Monitoring Layer
        B[Prometheus Scraper]
        C[Grafana Dashboard]
        D[Alertmanager]
    end
    A -- expose metrics --> B
    B -- store & query --> C
    B -- trigger alert --> D

Step 1: Adding a Health Check Endpoint

A health check can be simple—for example, in a Node.js Express application:

 1// app.js (Node.js/Express)
 2const express = require('express');
 3const app = express();
 4
 5app.get('/healthz', (req, res) => {
 6  // Check critical dependencies (e.g., DB connection)
 7  let dbHealthy = checkDBConnection();
 8  if(dbHealthy) {
 9      res.status(200).json({ status: 'ok' });
10  } else {
11      res.status(500).json({ status: 'error' });
12  }
13});
14
15// Dummy utility
16function checkDBConnection() {
17  // Simulate a random result (healthy or not)
18  return Math.random() > 0.05;
19}

Step 2: Exporting the Health Status as a Prometheus Metric

With the help of the prom-client library (for Node.js), a custom metric can be added:

 1// Monitoring integration
 2const client = require('prom-client');
 3const register = new client.Registry();
 4
 5const healthGauge = new client.Gauge({ 
 6  name: 'orderapi_health_status', 
 7  help: 'Current health check status: 1=UP, 0=DOWN'
 8});
 9register.registerMetric(healthGauge);
10
11app.get('/healthz', async (req, res) => {
12  let status = checkDBConnection() ? 1 : 0;
13  healthGauge.set(status);
14  res.status(status ? 200 : 500).json({ status: status ? 'ok' : 'error' });
15});
16
17app.get('/metrics', async (req, res) => {
18   res.set('Content-Type', register.contentType);
19   res.end(await register.metrics());
20});

Now, the /metrics endpoint exposes the following Prometheus metric:

1# HELP orderapi_health_status Current health check status: 1=UP, 0=DOWN
2# TYPE orderapi_health_status gauge
3orderapi_health_status 1

Step 3: Scraping via Prometheus

Add a static job configuration to the prometheus.yml file:

1scrape_configs:
2  - job_name: 'order-api-health'
3    metrics_path: /metrics
4    static_configs:
5      - targets: ['order-api.prod.svc.cluster.local:3000']

With this, Prometheus will scrape /metrics on every interval (15 seconds by default).

Step 4: Building an Alert in Prometheus

Add an alert rule, for example in alerts.yml:

 1groups:
 2- name: order-api.rules
 3  rules:
 4  - alert: OrderAPIHealthDown
 5    expr: orderapi_health_status == 0
 6    for: 5m
 7    labels:
 8      severity: critical
 9    annotations:
10      summary: "Order API Health Check DOWN"
11      description: "order-api has been unhealthy for 5 consecutive minutes"

This alert will fire if the health status drops (0) for more than 5 minutes.

Step 5: Displaying the Status in Grafana

A simple query in Grafana:

1orderapi_health_status

The table below is a simulation of the monitoring results displayed by Grafana:

Step 6: Simulating a Health Check DOWN Incident

To test the resilience of monitoring and alerting:

1. Modify the checkDBConnection() function so it always returns false.
2. After a few intervals, the metric value will drop to 0.
3. The Prometheus alert will fire.
4. Alertmanager sends a notification to the ops channel (e.g., Slack/Email).

Incident response sequence diagram:

sequenceDiagram
    participant Service as order-api
    participant Prometheus
    participant Alertmanager
    participant DevOps

    Service->>Prometheus: Expose metrics (health 0)
    Prometheus->>Prometheus: Evaluate alert rules
    Prometheus->>Alertmanager: Fire alert (OrderAPIHealthDown)
    Alertmanager->>DevOps: Notify via Slack/Email
    DevOps->>Service: Investigate & recover

sequenceDiagram
    participant Service as order-api
    participant Prometheus
    participant Alertmanager
    participant DevOps

    Service->>Prometheus: Expose metrics (health 0)
    Prometheus->>Prometheus: Evaluate alert rules
    Prometheus->>Alertmanager: Fire alert (OrderAPIHealthDown)
    Alertmanager->>DevOps: Notify via Slack/Email
    DevOps->>Service: Investigate & recover

Observations and Best Practices

1. Choose the Right Health Check

Make sure the health check actually touches your main dependencies (DB, Redis, Storage, etc.), not just returning 200 OK without any real validation.

2. Monitor Response Time

Other metrics such as health check duration are important:

1const healthLatency = new client.Gauge({ name: 'orderapi_health_latency', help: 'Health check duration in ms' });
2register.registerMetric(healthLatency);
3
4app.get('/healthz', async (req, res) => {
5  let t0 = Date.now();
6  // (...health checks...)
7  healthLatency.set(Date.now() - t0);
8  // ...
9});

3. Recovery Workflow

Alerts must be actionable (not spammy), and the DevOps team must have a quick recovery SOP.

Conclusion

Health check monitoring with Prometheus isn’t just about ticking a “compliance checklist”—it’s the foundation for maintaining reliability in production. With this process—from exposing the endpoint to alerting—the engineering team can react quickly before customers are affected.

This case study is just a baseline. In a real-world implementation, consider endpoint security, scrape tuning, multi-instance setups, and smarter alerting (e.g., auto-remediation). I hope this sharing is useful for your production environment!

How does health check monitoring work in your team? Share your thoughts in the comments!