# Observability in Your Application: Metrics with OpenTelemetry

Understanding an application's behavior in real-time is essential when deploying it. Think of it as checking the weather before heading out—you want to know the state of your app at a glance.

Key indicators such as **requests per minute (RPM)**, **resource usage**, **response times**, and **custom business metrics** are invaluable. Metrics give you this insight, enabling you to monitor and optimize your application's performance.

In this post, we’ll explore using OpenTelemetry (Otel) metrics and set up a pipeline to visualize them in Grafana.

---

### **OpenTelemetry Metrics: Key Concepts**

![](https://embed.filekitcdn.com/e/nXeSuArHTLg8jXaizBHT2T/3Z5fjza4v6gV8aHzJDQmv6/email align="left")

Before diving into implementation, let’s understand the basic terms in OpenTelemetry Metrics:

* **MeterProvider**: A singleton object that manages all meters in your application. Think of it as the provider for meters.
    
* **Meters**: Groups of instruments used to collect related metrics. For example, you might use one meter per microservice or domain.
    
* **Instrument**: A specific type of metric, such as counters, histograms, or gauges.
    
* **Metric**: The actual data collected. For example, incrementing a counter when an endpoint is called represents a metric.
    

These are the foundational concepts you need to work with OpenTelemetry metrics.

---

### **Types of Metrics**

OpenTelemetry supports three primary types of metrics:

1. **Synchronous Metrics**: These are recorded at the moment an event occurs. For example, tracking the number of HTTP requests.
    
2. **Additive Metrics**: Instruments like `UpDownCounter` track values that can go up and down, such as active users.
    
3. **Monotonic Metrics**: A subset of additive metrics that only increase over time, like total orders placed.
    

---

### **Setting Up Metrics: A Practical Example**

![](https://embed.filekitcdn.com/e/nXeSuArHTLg8jXaizBHT2T/3swZjuX3iHaYgzoUPpedCH/email align="left")

We’ll implement metrics for an **Order API** that:

1. Creates orders.
    
2. Ships them.
    
3. Retrieves them by ID.
    

We’ll track:

* Requests per minute (RPM).
    
* Response times.
    
* Custom metrics, like the number of shipped and canceled orders in the past week.
    

Our setup includes **OpenTelemetry**, **Prometheus**, and **Grafana**:

1. **OpenTelemetry** collects metrics.
    
2. **Prometheus** scrapes and stores metrics.
    
3. **Grafana** visualizes them in dashboards.
    
    ![](https://embed.filekitcdn.com/e/nXeSuArHTLg8jXaizBHT2T/gr4hgjbE1C6XnQhidSSTQU/email align="center")
    

---

### **Code Implementation**

#### Initializing a MeterProvider

First, initialize the MeterProvider:

```go
func InitMeter() (*metric2.MeterProvider, error) {
    ctx := context.Background()
    metricExporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithEndpoint("localhost:4318"), otlpmetrichttp.WithInsecure())
    if err != nil {
        return nil, err
    }

    meterProvider := metric2.NewMeterProvider(
        metric2.WithReader(metric2.NewPeriodicReader(metricExporter)),
        metric2.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("Order Api"),
        )),
    )
    otel.SetMeterProvider(meterProvider)

    return meterProvider, nil
}
```

This creates a global object for your app to use. Add this to your app’s initialization:

```go
mp, err := trace.InitMeter()
if err != nil {
    log.Fatal("Failed to initialize meter provider")
}
defer func() { _ = mp.Shutdown(context.Background()) }()
```

#### Exporting Metrics with FiberPrometheus

We’ll use **FiberPrometheus** middleware to export standard HTTP metrics like:

* `http_requests_total`
    
* `http_request_duration_seconds`
    

```go
prometheus := fiberprometheus.New("order-api")
prometheus.RegisterAt(app, "/metrics")
app.Use(prometheus.Middleware)
```

This will enable Prometheus to scrape metrics from the `/metrics` endpoint.

---

**Adding Custom Metrics: Shipped Orders**

For custom metrics, like shipped orders, define and register them:

```go
func InitMetrics(db *gorm.DB) {
    meterProvider := otel.GetMeterProvider()
    meter := meterProvider.Meter("Order Api")
    initShippedOrdersMetric(db, meter)
}

func initShippedOrdersMetric(db *gorm.DB, meter metric.Meter) error {
    shippedOrdersMetric, err := meter.Int64ObservableGauge(
        "shipped_orders_count_metric",
        metric.WithDescription("Counts the number of shipped orders"),
    )
    if err != nil {
        log.Fatalf("Failed to create metric: %v", err)
    }

    _, err = meter.RegisterCallback(
        func(ctx context.Context, observer metric.Observer) error {
            var count int64
            result := db.WithContext(ctx).Model(&entity.Order{}).
                Where("is_shipped = ? AND created_at > NOW() - INTERVAL '7 DAYS'", true).
                Count(&count)
            if result.Error != nil {
                log.Printf("Database query error: %v", result.Error)
            }

            observer.ObserveInt64(shippedOrdersMetric, count)
            return nil
        },
        shippedOrdersMetric,
    )
    return err
}
```

This captures the count of shipped orders in the past 7 days and exports it as a custom metric.

---

**Visualizing Metrics in Grafana**

Once Prometheus scrapes the metrics, you can visualize them in Grafana.

#### Setting Up the Dashboard

1. Open Grafana at [`http://localhost:3000`](http://localhost:3000) and log in.
    
2. Create a new dashboard from the sidebar.
    

#### Example Visualizations

* **Requests per Minute (RPM)**:
    
* ```plaintext
    sum(rate(http_requests_total[1m])) by (job) * 60
    ```
    
    ![](https://embed.filekitcdn.com/e/nXeSuArHTLg8jXaizBHT2T/sjvugSS2jhKNhFWSMXuqDv/email align="left")
    
    Use the "Time Series" visualization type.
    
* **95th Percentile Response Time**:
    
    ![](https://embed.filekitcdn.com/e/nXeSuArHTLg8jXaizBHT2T/xfQMhuU6AUyJTyqAZLpH3c/email align="left")
    
    ```plaintext
    histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[1m])) by (le))
    ```
    
* **Shipped Orders (Custom Metric)**: For the `shipped_orders_count_metric`, use a single-value panel to display the current count.  
    
    ![](https://embed.filekitcdn.com/e/nXeSuArHTLg8jXaizBHT2T/itkeyQ4SAxqdfDmjJLtwdn/email align="left")
    

---

### **Conclusion**

With OpenTelemetry, Prometheus, and Grafana, you can track key performance metrics and custom business data in real time. Metrics like RPM, response times, and shipped orders help ensure your app runs smoothly and efficiently.

The complete codebase and setup instructions are available on GitHub. Explore and adapt it to your app for enhanced observability!

If you want to learn more about Golang and backend development, subscribe to my newsletter at firatkomurcu.com

May the force be with you!