Prompt Versioning

Prompt Versioning and Management

Why Version Prompts?
Prompts are code. Track changes, roll back issues, and collaborate effectively.

Git-Based Versioning

Metadata Format

# metadata.yaml
name: customer_support_main
version: 2.3.1
author: team-ai
created: 2024-01-15
updated: 2024-03-20

model_requirements:
  min_context: 4096
  recommended_model: gpt-4o

evaluation:
  last_eval_date: 2024-03-18
  accuracy: 0.94
  latency_p50: 1.2s

Prompt Template Management

from jinja2 import Environment, FileSystemLoader

env = Environment(loader=FileSystemLoader("prompts"))

def load_prompt(name, version=None, **kwargs):
    if version:
        path = f"{name}/v{version}/prompt.md"
    else:
        path = f"{name}/prompt.md"

    template = env.get_template(path)
    return template.render(**kwargs)

# Usage
prompt = load_prompt("summarization", version=2, max_length=100)

LangSmith/LangChain Hub

from langchain import hub

# Push prompt
hub.push("my-org/customer-support-v2", prompt_template)

# Pull prompt
prompt = hub.pull("my-org/customer-support-v2")

A/B Testing Prompts

import random

class PromptExperiment:
    def __init__(self, prompts, weights=None):
        self.prompts = prompts
        self.weights = weights or [1/len(prompts)] * len(prompts)

    def get_prompt(self, user_id):
        # Deterministic assignment based on user
        bucket = hash(user_id) % 100
        cumulative = 0
        for prompt, weight in zip(self.prompts, self.weights):
            cumulative += weight * 100
            if bucket < cumulative:
                return prompt
        return self.prompts[-1]

Prompt Registry

class PromptRegistry:
    def __init__(self, storage):
        self.storage = storage

    def register(self, name, prompt, version, metadata):
        key = f"{name}:{version}"
        self.storage.set(key, {
            "prompt": prompt,
            "metadata": metadata,
            "created_at": datetime.now()
        })

    def get(self, name, version="latest"):
        if version == "latest":
            version = self.get_latest_version(name)
        return self.storage.get(f"{name}:{version}")

Best Practices
- Use semantic versioning (major.minor.patch)
- Include evaluation metrics with versions
- Document changes in changelog
- Test prompts before deploying
- Keep production prompts immutable
- A/B test significant changes