Instrumentation

Instrumentation is the process of adding code to your application to expose metrics that Prometheus can scrape. The PCA exam tests your understanding of metric types and best practices.

Metric Types

Prometheus defines four core metric types:

Counter

A counter is a cumulative metric that only goes up (or resets to zero on restart).

Use cases: request counts, errors, bytes sent

httpRequests := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total number of HTTP requests.",
    },
    []string{"method", "status"},
)

// Increment
httpRequests.WithLabelValues("GET", "200").Inc()
httpRequests.WithLabelValues("POST", "500").Add(1)

Naming convention: Use _total suffix for counters.

Gauge

A gauge represents a value that can go up and down.

Use cases: temperature, memory usage, active connections

activeConns := prometheus.NewGauge(prometheus.GaugeOpts{
    Name: "active_connections",
    Help: "Number of active connections.",
})

// Set, increment, decrement
activeConns.Set(42)
activeConns.Inc()
activeConns.Dec()
activeConns.Add(5)
activeConns.Sub(2)

Histogram

A histogram samples observations and counts them in configurable buckets. It also provides a sum and count of observations.

Use cases: request durations, response sizes

requestDuration := prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request duration in seconds.",
        Buckets: prometheus.DefBuckets, // .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10
    },
    []string{"method"},
)

// Observe a value
requestDuration.WithLabelValues("GET").Observe(0.42)

A histogram exposes three time series:

• http_request_duration_seconds_bucket{le="0.5"} — count of observations <= 0.5s
• http_request_duration_seconds_sum — total sum of observed values
• http_request_duration_seconds_count — total count of observations

Summary

A summary calculates streaming quantiles on the client side.

Use cases: similar to histograms, but when you need pre-calculated quantiles

requestDuration := prometheus.NewSummary(prometheus.SummaryOpts{
    Name:       "http_request_duration_seconds",
    Help:       "HTTP request duration in seconds.",
    Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001},
})

Histogram vs Summary

Recommendation: Prefer histograms in most cases. They can be aggregated across instances, and histogram_quantile() in PromQL is flexible.

Exposition Format

Metrics are exposed in a text-based format:

# HELP http_requests_total Total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1234
http_requests_total{method="POST",status="201"} 56

# HELP active_connections Number of active connections.
# TYPE active_connections gauge
active_connections 42

Key rules:

• # HELP describes the metric
• # TYPE declares the metric type
• Label values are double-quoted
• Metric names follow the pattern [a-zA-Z_:][a-zA-Z0-9_:]*

Naming Conventions

Follow these conventions for metric names:

1. Use the application or library name as prefix: myapp_requests_total
2. Use snake_case for names
3. Include the unit as suffix: _seconds, _bytes, _total
4. Counters must end in _total
5. Use base units (seconds not milliseconds, bytes not kilobytes)

Client Libraries

Prometheus provides official client libraries for:

• Go — github.com/prometheus/client_golang
• Java — io.prometheus:simpleclient
• Python — prometheus_client
• Ruby — prometheus-client

Each library provides:

• Metric type constructors (Counter, Gauge, Histogram, Summary)
• A default HTTP handler to expose metrics
• A default registry for metric collection

Key Exam Tips

1. Counter vs Gauge: If the value can decrease, use a gauge. If it only ever goes up, use a counter.
2. _total suffix: Required for counters in the OpenMetrics format.
3. Default buckets: Know that prometheus.DefBuckets provides sensible defaults for HTTP latency.
4. Label cardinality: Avoid high-cardinality labels (like user IDs). Each unique label combination creates a new time series.
5. info metrics: Use _info suffix for metrics that expose textual information as labels with a constant value of 1.