Metrics Collection

Metrics collection is the practice of systematically gathering numerical measurements about system health, performance, and behavior at regular intervals. In AI/ML systems, metrics provide the quantitative foundation for monitoring, alerting, capacity planning, and optimization.

Types of Metrics

• Counter: A monotonically increasing value — total requests served, total tokens generated, total errors. Can only go up (or reset to zero).
• Gauge: A value that can go up or down — current GPU utilization, active connections, memory usage, queue depth.
• Histogram: Distribution of values — request latency distribution, token count distribution. Enables percentile calculations (p50, p95, p99).
• Summary: Pre-computed percentiles over a sliding time window — similar to histograms but computed on the client side.

Key Metrics for AI Systems

• Inference Latency: Time to first token (TTFT), time per output token (TPOT), and total generation time.
• Throughput: Requests per second, tokens per second.
• GPU Utilization: Percentage of GPU compute capacity in use.
• GPU Memory: VRAM usage, KV cache size, available memory.
• Error Rates: By error type (timeout, rate limit, model error, safety filter).
• Queue Depth: Number of pending requests waiting for inference.
• Token Usage: Input/output tokens per request for cost tracking.
• Model Quality: Online quality scores, user ratings, task completion rates.

Collection Architecture

• Push Model: Application pushes metrics to a central collector (StatsD, Datadog Agent). Lower latency, application controls send timing.
• Pull Model: Collector scrapes metrics from application endpoints (Prometheus). Simpler application code, collector controls timing.
• Hybrid: OpenTelemetry supports both push and pull, with protocol translation.

Tools

• Prometheus: Pull-based, time-series database with powerful query language (PromQL). Industry standard for Kubernetes.
• Datadog: SaaS metrics platform with AI-specific integrations.
• CloudWatch / Cloud Monitoring: Cloud-native metrics from AWS/GCP.
• OpenTelemetry: Vendor-neutral metrics collection SDK and protocol.

Metrics collection is the quantitative backbone of observability — without metrics, you're operating blind on system health and performance.