Observability in Your Application: Metrics with OpenTelemetry

Understanding an application's behavior in real-time is essential when deploying it. Think of it as checking the weather before heading out—you want to know the state of your app at a glance.

Key indicators such as requests per minute (RPM), resource usage, response times, and custom business metrics are invaluable. Metrics give you this insight, enabling you to monitor and optimize your application's performance.

In this post, we’ll explore using OpenTelemetry (Otel) metrics and set up a pipeline to visualize them in Grafana.

---

OpenTelemetry Metrics: Key Concepts

Before diving into implementation, let’s understand the basic terms in OpenTelemetry Metrics:

• MeterProvider: A singleton object that manages all meters in your application. Think of it as the provider for meters.

• Meters: Groups of instruments used to collect related metrics. For example, you might use one meter per microservice or domain.

• Instrument: A specific type of metric, such as counters, histograms, or gauges.

• Metric: The actual data collected. For example, incrementing a counter when an endpoint is called represents a metric.

These are the foundational concepts you need to work with OpenTelemetry metrics.

---

Types of Metrics

OpenTelemetry supports three primary types of metrics:

1. Synchronous Metrics: These are recorded at the moment an event occurs. For example, tracking the number of HTTP requests.

2. Additive Metrics: Instruments like UpDownCounter track values that can go up and down, such as active users.

3. Monotonic Metrics: A subset of additive metrics that only increase over time, like total orders placed.

---

Setting Up Metrics: A Practical Example

We’ll implement metrics for an Order API that:

1. Creates orders.

2. Ships them.

3. Retrieves them by ID.

We’ll track:

• Requests per minute (RPM).

• Response times.

• Custom metrics, like the number of shipped and canceled orders in the past week.

Our setup includes OpenTelemetry, Prometheus, and Grafana:

1. OpenTelemetry collects metrics.

2. Prometheus scrapes and stores metrics.

3. Grafana visualizes them in dashboards.

   

---

Code Implementation

Initializing a MeterProvider

First, initialize the MeterProvider:

func InitMeter() (*metric2.MeterProvider, error) {
    ctx := context.Background()
    metricExporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithEndpoint("localhost:4318"), otlpmetrichttp.WithInsecure())
    if err != nil {
        return nil, err
    }

    meterProvider := metric2.NewMeterProvider(
        metric2.WithReader(metric2.NewPeriodicReader(metricExporter)),
        metric2.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("Order Api"),
        )),
    )
    otel.SetMeterProvider(meterProvider)

    return meterProvider, nil
}

This creates a global object for your app to use. Add this to your app’s initialization:

mp, err := trace.InitMeter()
if err != nil {
    log.Fatal("Failed to initialize meter provider")
}
defer func() { _ = mp.Shutdown(context.Background()) }()

Exporting Metrics with FiberPrometheus

We’ll use FiberPrometheus middleware to export standard HTTP metrics like:

• http_requests_total

• http_request_duration_seconds

prometheus := fiberprometheus.New("order-api")
prometheus.RegisterAt(app, "/metrics")
app.Use(prometheus.Middleware)

This will enable Prometheus to scrape metrics from the /metrics endpoint.

---

Adding Custom Metrics: Shipped Orders

For custom metrics, like shipped orders, define and register them:

func InitMetrics(db *gorm.DB) {
    meterProvider := otel.GetMeterProvider()
    meter := meterProvider.Meter("Order Api")
    initShippedOrdersMetric(db, meter)
}

func initShippedOrdersMetric(db *gorm.DB, meter metric.Meter) error {
    shippedOrdersMetric, err := meter.Int64ObservableGauge(
        "shipped_orders_count_metric",
        metric.WithDescription("Counts the number of shipped orders"),
    )
    if err != nil {
        log.Fatalf("Failed to create metric: %v", err)
    }

    _, err = meter.RegisterCallback(
        func(ctx context.Context, observer metric.Observer) error {
            var count int64
            result := db.WithContext(ctx).Model(&entity.Order{}).
                Where("is_shipped = ? AND created_at > NOW() - INTERVAL '7 DAYS'", true).
                Count(&count)
            if result.Error != nil {
                log.Printf("Database query error: %v", result.Error)
            }

            observer.ObserveInt64(shippedOrdersMetric, count)
            return nil
        },
        shippedOrdersMetric,
    )
    return err
}

This captures the count of shipped orders in the past 7 days and exports it as a custom metric.

---

Visualizing Metrics in Grafana

Once Prometheus scrapes the metrics, you can visualize them in Grafana.

Setting Up the Dashboard

1. Open Grafana at http://localhost:3000 and log in.

2. Create a new dashboard from the sidebar.

Example Visualizations

• Requests per Minute (RPM):

•   sum(rate(http_requests_total[1m])) by (job) * 60
  

  

  Use the "Time Series" visualization type.

• 95th Percentile Response Time:

  

    histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[1m])) by (le))
  

• Shipped Orders (Custom Metric): For the shipped_orders_count_metric, use a single-value panel to display the current count.

  

---

Conclusion

With OpenTelemetry, Prometheus, and Grafana, you can track key performance metrics and custom business data in real time. Metrics like RPM, response times, and shipped orders help ensure your app runs smoothly and efficiently.

The complete codebase and setup instructions are available on GitHub. Explore and adapt it to your app for enhanced observability!

If you want to learn more about Golang and backend development, subscribe to my newsletter at firatkomurcu.com

May the force be with you!