Grafana
Grafana is the open-source observability platform that connects to multiple data sources and renders unified dashboards for metrics, logs, and traces — serving as the "single pane of glass" that teams use to visualize AI infrastructure health, model performance, GPU utilization, and LLM cost analytics without storing data itself.
What Is Grafana?
- Definition: A multi-source visualization platform that queries data from Prometheus, InfluxDB, Elasticsearch, Loki, Jaeger, PostgreSQL, and dozens of other backends — rendering interactive dashboards with graphs, heatmaps, tables, and alerts.
- Architecture: Grafana is a pure visualization layer — it does not store metrics or logs. It queries existing data stores and renders results, making it composable with any monitoring stack.
- Created By: Torkel Odegaard (2014), originally forked from Kibana. Now maintained by Grafana Labs with a massive open-source community.
- Scale: Used by Netflix, Uber, PayPal, and virtually every major tech company — pre-built dashboards available for every popular AI framework and GPU monitoring stack.
Why Grafana Matters for AI Teams
- Training Run Monitoring: Visualize loss curves, gradient norms, learning rate schedules, and GPU utilization side-by-side in real time during model training.
- Inference Dashboard: Track TTFT (Time to First Token), tokens per second, queue depth, error rates, and cost per query with automatic alerting.
- GPU Fleet Management: Monitor temperature, memory usage, power draw, and SM utilization across hundreds of GPUs simultaneously — spot thermal throttling and underutilization instantly.
- Multi-Source Correlation: Overlay application metrics (Prometheus), logs (Loki), and traces (Tempo/Jaeger) on the same timeline — find root causes by correlating a latency spike with a log error and a specific trace.
- Cost Analytics: Track OpenAI API costs, RunPod GPU hours, and inference infrastructure costs — visualize cost per user, per model, per feature.
Core Concepts
Data Sources: Grafana's connectivity layer. Configure once, query anywhere:
- Prometheus (metrics time-series)
- Loki (logs — Prometheus-like, but for log streams)
- Tempo (distributed traces)
- InfluxDB (time-series)
- PostgreSQL / MySQL (structured data — query your experiment tracking DB)
- CloudWatch, Azure Monitor, Google Cloud Monitoring
- Elasticsearch / OpenSearch (log search and analytics)
Panels: Individual visualization units within a dashboard:
- Time Series: Line/bar charts for metrics over time.
- Stat: Single big number — current GPU temp, error rate, queue depth.
- Table: Tabular data — top 10 slowest queries, highest-cost models.
- Heatmap: Distribution over time — request latency distribution visualized as a heatmap.
- Logs Panel: Streaming log viewer filtered by labels.
- Traces Panel: Flame graph visualization of distributed traces.
Dashboards: Collections of panels arranged on a grid. Shareable as JSON — import community dashboards from grafana.com/grafana/dashboards.
Alerting: Grafana Alerting evaluates queries on a schedule and sends notifications via Slack, PagerDuty, email, and webhooks when thresholds are breached.
Pre-Built AI/ML Dashboards
| Dashboard | Source | Key Panels |
|---|---|---|
| NVIDIA DCGM | grafana.com (ID 12239) | GPU util, temp, memory per device |
| Kubernetes cluster | grafana.com (ID 15661) | Pod health, resource usage |
| vLLM Inference | vLLM docs | TTFT, throughput, queue, KV cache |
| W&B alternative | Custom | Training loss, eval metrics |
| Node Exporter Full | grafana.com (ID 1860) | CPU, memory, disk, network |
Grafana Stack (LGTM)
Grafana Labs provides a full open-source observability stack:
- Loki — Log aggregation (like Prometheus but for logs).
- Grafana — Visualization layer.
- Tempo — Distributed tracing backend.
- Mimir — Long-term metrics storage (horizontally scalable Prometheus).
Together these four components cover all three observability pillars (metrics, logs, traces) in a single integrated stack.
Practical AI Inference Dashboard
A production LLM serving dashboard typically includes:
- TTFT p50/p95/p99 over time (line chart).
- Tokens per second by model (stacked bar).
- Active requests in queue (gauge).
- GPU memory utilization per device (multi-line).
- Error rate by error type (bar chart).
- Cost per 1K tokens trend (time series).
- Top 10 longest prompts by user (table).
Grafana is the universal lens through which AI teams observe their systems — its ability to unify metrics, logs, and traces from any data source into a single, interactive view makes it indispensable for monitoring the full stack from GPU hardware to LLM response quality in production.