39 Monitoring and Tracing (Prometheus & OpenTelemetry)

In today’s cloud native era, distributed systems have become the daily bread of engineers. Microservices, Kubernetes, auto-scaling, and CI/CD make systems increasingly dynamic, but they also add complexity. Even a simple question like “Why is my service slow?” can turn into a mysterious debugging adventure.

Monitoring and tracing are two essential tools that are truly crucial for keeping your application healthy, scalable, and reliable. In this article, I want to share how Prometheus and OpenTelemetry can become the backbone solution for modern monitoring and tracing, complete with real code, simulations, and best practices.

1. Monitoring vs. Tracing: What’s the Difference?

Monitoring is the process of collecting, analyzing, and visualizing metrics from a system—such as CPU, memory, latency, error rate, and so on. The goal is to make us aware when something is wrong, so we can act quickly and observe performance trends.

Tracing is the process of tracking the execution flow of a request, usually from one service to another in an end-to-end manner. This is extremely powerful for finding bottlenecks and debugging distributed systems.

2. Prometheus: Powerful, Open Source Monitoring

Prometheus is a highly popular tool for monitoring time-series metrics. It “pulls” data from your application periodically, then stores it for visualization or alerting.

2.1 Prometheus Architecture (Simplified)

flowchart LR
    subgraph A[Application]
        Svc1[Microservice 1] -->|Exposes /metrics| M[Prometheus]
        Svc2[Microservice 2] -->|Exposes /metrics| M
    end
    M -->|Time series| Storage[(Prometheus TSDB)]
    M -->|Monitoring| Grafana
    M -->|Alert| Alertmanager

flowchart LR
    subgraph A[Application]
        Svc1[Microservice 1] -->|Exposes /metrics| M[Prometheus]
        Svc2[Microservice 2] -->|Exposes /metrics| M
    end
    M -->|Time series| Storage[(Prometheus TSDB)]
    M -->|Monitoring| Grafana
    M -->|Alert| Alertmanager

1. Application: Each microservice exposes a /metrics endpoint (format: Prometheus exposition format).
2. Prometheus Server: Regularly scrapes this endpoint and stores the metrics.
3. Visualization & Alerts: The data can be used by Grafana or other dashboards, or triggered via Alertmanager when an anomaly occurs.

2.2 Code Example: Exposing Metrics in Go

Suppose we have a simple order service.

 1package main
 2
 3import (
 4    "net/http"
 5    "github.com/prometheus/client_golang/prometheus"
 6    "github.com/prometheus/client_golang/prometheus/promhttp"
 7)
 8
 9var (
10    orderCounter = prometheus.NewCounter(prometheus.CounterOpts{
11        Name: "order_created_total",
12        Help: "Number of orders successfully created",
13    })
14)
15
16func init() {
17    prometheus.MustRegister(orderCounter)
18}
19
20func orderHandler(w http.ResponseWriter, r *http.Request) {
21    // some order logic...
22    orderCounter.Inc()
23    w.WriteHeader(http.StatusOK)
24}
25
26func main() {
27    http.Handle("/metrics", promhttp.Handler())
28    http.HandleFunc("/order", orderHandler)
29    http.ListenAndServe(":8080", nil)
30}

Explanation:

• When the /order endpoint is hit, the order_created_total counter increases.
• Prometheus simply needs to configure the scrape: <host>:8080/metrics.

3. OpenTelemetry: The New Standard for Distributed Tracing

End-to-end tracing in microservices cannot rely on standard logs alone. OpenTelemetry (OTel) emerged as a vendor-neutral toolkit for tracing, metrics, and logs. It became the “de facto standard” after the merger of the OpenTracing and OpenCensus projects.

3.1 OpenTelemetry Architecture

flowchart TD
    UserReq(User Request)
    A[Service A] -->|span context| B[Service B]
    B -->|span context| C[Service C]
    A --> OTEL(OTel Collector)
    B --> OTEL
    C --> OTEL
    OTEL --> Jaeger

flowchart TD
    UserReq(User Request)
    A[Service A] -->|span context| B[Service B]
    B -->|span context| C[Service C]
    A --> OTEL(OTel Collector)
    B --> OTEL
    C --> OTEL
    OTEL --> Jaeger

1. Each service defines a span and forwards the context to the next service.
2. OpenTelemetry Collector is the agent that collects, processes, and then forwards tracing data to a backend (Jaeger, Zipkin, etc.).

3.2 Code Exploration: Implementing Tracing in Go

Install the dependencies:

1go get go.opentelemetry.io/otel
2go get go.opentelemetry.io/otel/sdk
3go get go.opentelemetry.io/otel/exporters/jaeger

Code snippet:

 1import (
 2    "context"
 3    "go.opentelemetry.io/otel"
 4    "go.opentelemetry.io/otel/trace"
 5)
 6
 7func processOrder(ctx context.Context) {
 8    tracer := otel.Tracer("order-service")
 9    ctx, span := tracer.Start(ctx, "ProcessOrder")
10    defer span.End()
11
12    // ... some process ...
13    inventoryCheck(ctx)
14}
15
16func inventoryCheck(ctx context.Context) {
17    tracer := otel.Tracer("order-service")
18    _, span := tracer.Start(ctx, "InventoryCheck")
19    defer span.End()
20
21    // ... some inventory call ...
22}

Essentially, every request/request handler forms a span hierarchy (parent-child). This is what then gets visualized as a trace diagram (waterfall) in Jaeger.

4. Simulation: Observing the System from Two Sides

We want to answer the question: “How fast is an order processed? If it’s slow, where is the bottleneck?”

1. With Prometheus, we can observe the incoming order rate, the error rate, and even the processing duration through metrics such as a histogram:
   • order_created_total, order_error_total, order_processing_seconds_bucket
2. With OpenTelemetry, we can see the time breakdown of each handler, service-to-service calls, and the root cause of a slow request.

5. Comparing Usage: When to Use Which?

Takeaway:

• Use Prometheus for system health at a macro level and alerting.
• Use OpenTelemetry/Tracing for root-cause analysis of individual requests.

6. Integration and Best Practice Recommendations

• Always monitor basic service metrics (rate, error rate, latency).
• Add tracing on critical paths (transactions, auth, etc.).
• Integrate logs, metrics, and traces so troubleshooting becomes faster. Many modern tools (Grafana, Tempo/Jaeger) already support correlation IDs.
• Run load tests periodically and validate whether traces can actually catch the real “bottleneck,” not just noise.

7. Conclusion

Monitoring and tracing are not just debugging tools, but recorders of modern business health. In observability for the cloud native era, Prometheus and OpenTelemetry are the foundation, not an add-on feature.

References

• Prometheus Documentation
• OpenTelemetry Docs
• Jaeger Tracing

I hope your observability insight grows sharper and your troubleshooting becomes more productive 🎯