Analytics

AI analytics and usage metrics involve tracking and analyzing how AI features are used within products — measuring query patterns, performance characteristics, user engagement, and quality indicators to optimize AI capabilities, control costs, and demonstrate value to stakeholders.

Why AI Analytics Matter

Optimization: Identify slow or expensive queries.
Quality: Detect degradation in responses.
Cost Control: Understand and optimize spend.
ROI: Demonstrate AI feature value.
Planning: Capacity and scaling decisions.

Key Metrics Categories

Usage Metrics:

Metric                | What It Measures
----------------------|----------------------------------
Query Volume          | Total requests over time
Active Users          | Unique users using AI features
Queries per User      | Engagement depth
Feature Adoption      | % of users trying AI features
Session Patterns      | When/how AI is used

Performance Metrics:

Metric                | What It Measures
----------------------|----------------------------------
Latency (P50/P95/P99) | Response time distribution
TTFT                  | Time to first token (streaming)
Throughput            | Requests/sec capacity
Error Rate            | Failed requests percentage
Timeout Rate          | Requests exceeding limit

Quality Metrics:

Metric                | What It Measures
----------------------|----------------------------------
User Ratings          | Explicit feedback (thumbs up/down)
Completion Rate       | Users accepting AI output
Edit Rate             | How much users modify output
Regeneration Rate     | Users requesting new response
Task Success          | Goal completion with AI

Cost Metrics:

Metric                | What It Measures
----------------------|----------------------------------
Tokens per Query      | Input + output tokens
Cost per Query        | $ spent per request
Cost per User         | Monthly per-user AI spend
Model Distribution    | Which models serve what
Cache Hit Rate        | Savings from caching

Implementation

Basic Logging:

import time
import logging

class AIMetrics:
    def log_request(self, request_id, model, prompt_tokens, 
                    completion_tokens, latency, success):
        logging.info({
            "event": "ai_request",
            "request_id": request_id,
            "model": model,
            "prompt_tokens": prompt_tokens,
            "completion_tokens": completion_tokens,
            "latency_ms": latency,
            "success": success,
            "timestamp": time.time()
        })

# Usage
metrics = AIMetrics()

start = time.time()
response = await llm.generate(prompt)
latency = (time.time() - start) * 1000

metrics.log_request(
    request_id=uuid.uuid4(),
    model="gpt-4o",
    prompt_tokens=response.usage.prompt_tokens,
    completion_tokens=response.usage.completion_tokens,
    latency=latency,
    success=True
)

Analytics Dashboard:

# SQL for daily metrics
"""
SELECT 
    DATE(timestamp) as date,
    COUNT(*) as total_queries,
    COUNT(DISTINCT user_id) as unique_users,
    AVG(latency_ms) as avg_latency,
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY latency_ms) as p95_latency,
    SUM(prompt_tokens + completion_tokens) as total_tokens,
    SUM(cost) as total_cost,
    AVG(CASE WHEN user_rating IS NOT NULL THEN user_rating END) as avg_rating
FROM ai_requests
WHERE timestamp > NOW() - INTERVAL '30 days'
GROUP BY DATE(timestamp)
ORDER BY date DESC
"""

Dashboards

Essential Views:

Dashboard          | Key Visuals
-------------------|----------------------------------
Usage Overview     | Query volume, active users, trends
Performance        | Latency distribution, errors
Cost               | Daily spend, cost per query
Quality            | Ratings, completion rate
Model Comparison   | Performance by model

Tools:

Tool              | Use Case
------------------|----------------------------------
Grafana           | Real-time dashboards
Datadog           | Full observability
Mixpanel          | Product analytics
LangSmith         | LLM-specific observability
Helicone          | LLM cost tracking
Custom            | Tailored to needs

Alerting

What to Alert On:

alerts = {
    "high_latency": {
        "condition": "p95_latency > 5000ms",
        "severity": "warning"
    },
    "error_rate": {
        "condition": "error_rate > 5%",
        "severity": "critical"
    },
    "cost_spike": {
        "condition": "hourly_cost > 2x average",
        "severity": "warning"
    },
    "quality_drop": {
        "condition": "rating_avg < 3.5",
        "severity": "warning"
    }
}

Best Practices

Log Everything: Can't analyze what you don't collect.
User Privacy: Anonymize/redact sensitive content.
Real-Time + Historical: Both immediate and trend analysis.
Correlate Metrics: Understand relationships.
Action-Oriented: Every dashboard should drive decisions.

AI analytics are essential for operating AI features responsibly — understanding usage, performance, and cost enables optimization, demonstrates value, and catches problems before users complain.

Analytics

Take analytics further