Prompt Ensemble

Prompt ensemble is the technique of combining predictions from multiple different prompts for the same task — leveraging the diversity of prompt formulations to produce more robust and accurate outputs than any single prompt alone.

Why Prompt Ensembles?

• LLM outputs are sensitive to prompt phrasing — different wordings of the same question can produce different answers.
• No single prompt is reliably optimal across all inputs — some prompts work better for certain examples.
• Ensembling reduces the variance of predictions by averaging out the idiosyncrasies of individual prompts.

Prompt Ensemble Methods

• Majority Voting: Run the same input through $k$ different prompts. Each prompt produces a prediction. The most common answer wins.
• Example for sentiment classification: 3 prompts say "positive," 2 say "negative" → final answer is "positive."
• Simple and effective for classification tasks.

• Weighted Voting: Assign weights to prompts based on their validation accuracy. Better prompts contribute more to the final decision.
• $\hat{y} = \arg\max_c \sum_i w_i \cdot \mathbb{1}[p_i = c]$

• Probability Averaging: Average the probability distributions over classes from each prompt. Choose the class with highest average probability.
• $P(c|x) = \frac{1}{K} \sum_{i=1}^{K} P_i(c|x)$
• Smoother than voting — uses confidence information.

• Verbalizer Ensemble: For classification, use multiple verbalizer mappings (e.g., "positive"/"negative" vs. "good"/"bad" vs. "favorable"/"unfavorable") and combine predictions.

Ensemble Prompt Diversity

• Template Diversity: Different instruction phrasings — "Classify this text," "Is this positive or negative?," "What sentiment does this express?"
• Format Diversity: Different output formats — "Answer with positive/negative," "Rate 1-5," "Explain and then classify."
• Perspective Diversity: Different reasoning angles — "As a movie critic...," "Consider the emotional tone...," "Focus on the factual claims..."
• Few-Shot Diversity: Different sets of demonstration examples — each prompt uses a different subset of few-shot examples.

Benefits

• Accuracy: Ensembles typically improve accuracy by 2–5% over the best single prompt.
• Robustness: Less sensitive to individual prompt failures or adversarial inputs.
• Reliability: More consistent performance across diverse inputs.
• Calibration: Ensemble probabilities tend to be better calibrated than single-prompt probabilities.

Costs

• Compute: Each prompt requires a separate model inference — $k$ prompts means $k×$ the compute cost.
• Latency: Sequential execution multiplies latency. Parallel execution helps but requires more resources.
• Diminishing Returns: Beyond 5–10 diverse prompts, additional prompts provide minimal improvement.

Prompt ensembles are one of the most reliable techniques for improving LLM accuracy — they exploit the observation that different prompts capture different aspects of the task, and combining them produces a more complete and robust understanding.