Demonstration Selection

Demonstration selection is the process of choosing the most effective in-context examples (demonstrations) to include in a few-shot prompt — because the quality, relevance, and composition of the examples significantly impacts the language model's performance on the target task.

Why Demonstration Selection Matters

• In few-shot learning, the model learns the task pattern from the provided examples — which examples are shown can change accuracy by 10–20% or more.
• Random selection may include irrelevant, redundant, or misleading examples.
• Strategic selection provides examples that are maximally informative for the specific input being processed.

Demonstration Selection Strategies

• Similarity-Based Selection: Choose examples most similar to the current test input.
• Embedding Similarity: Compute sentence embeddings for all candidate examples and the test input. Select the $k$ nearest neighbors by cosine similarity.
• Intuition: Similar examples demonstrate patterns most relevant to the current input — the model can more easily transfer the demonstrated pattern.
• Most widely used and consistently effective approach.

• Diversity-Based Selection: Choose examples that cover a wide range of the task space.
• Select examples from different categories, different difficulty levels, different patterns.
• Ensures the model sees the full scope of possible task behaviors.
• Works well when the test input distribution is unknown.

• Similarity + Diversity: Combine both — select examples that are relevant to the current input AND diverse among themselves.
• MMR (Maximal Marginal Relevance): Balance relevance to the query with diversity among selected examples.

• Difficulty-Based: Choose examples with moderate difficulty.
• Very easy examples may not be informative. Very hard or ambiguous examples may confuse the model.
• Select examples where the model has moderate confidence — most informative for learning.

• Label-Balanced Selection: Ensure the selected examples have a balanced distribution of labels/categories.
• Imbalanced demonstrations can bias the model toward over-represented classes.

Advanced Selection Methods

• Reinforcement Learning: Train a selector model that chooses demonstrations to maximize downstream task performance.
• Influence Functions: Estimate which training examples have the most positive influence on predicting the test input correctly.
• Iterative Selection: Use the model's initial prediction to refine example selection — if the model is uncertain, select more relevant examples and retry.

Practical Considerations

• Context Window: Limited context length means typically 3–10 examples fit — selection quality matters more than quantity.
• Example Format: Select examples that match the desired output format — the model imitates the demonstrated format.
• Recency: Examples positioned later in the prompt (closer to the test input) may have more influence than earlier ones.

Demonstration selection is one of the highest-impact prompt engineering techniques — systematic selection of few-shot examples can transform mediocre few-shot performance into state-of-the-art results.