Example Ordering

Example ordering (also called demonstration ordering) is the arrangement of in-context learning examples within a prompt to maximize model performance — because the order in which demonstrations are presented significantly affects how well the language model extracts and applies the task pattern.

Why Order Matters

• LLMs process text sequentially — the position of each example in the context creates different attention patterns and different inductive biases.
• Research shows that reordering the same examples can cause accuracy to vary by 10–15% or more — sometimes the difference between random and state-of-the-art performance.
• The model may give more weight to examples near the end of the prompt (recency bias) or near the beginning (primacy bias), depending on the model and task.

Ordering Effects

• Recency Bias: Many models weigh later examples more heavily — the last few demonstrations before the test input have outsized influence on the prediction.
• Primacy Bias: Some models (especially with shorter contexts) are more influenced by the first few examples.
• Label Bias: If the last several examples all have the same label, the model may be biased toward predicting that label for the test input.
• Pattern Recognition: Certain orderings make the task pattern more obvious to the model — for example, grouping similar examples together vs. alternating.

Ordering Strategies

• Random Ordering: Shuffle demonstrations randomly. Simple baseline, but suboptimal.
• Similarity-Based Ordering: Place the most similar example to the test input last (closest to the test input) — leverages recency bias to maximize the influence of the most relevant demonstration.
• Reverse Similarity: Place the most similar example first — works better for models with strong primacy bias.
• Difficulty Ordering: Arrange from easy to hard — starts with clear examples to establish the pattern, then shows more nuanced cases.
• Label Alternation: Alternate between different labels/categories — prevents label bias from consecutive same-label examples.
• Curriculum-Style: Start with diverse, representative examples and end with examples similar to the test input.

Optimal Ordering Methods

• Entropy-Based: Choose the ordering that minimizes the model's prediction entropy on a validation set — the ordering that makes the model most confident.
• Beam Search: Try multiple orderings and evaluate each — select the best. Computationally expensive but effective.
• Learned Ordering: Train a model to predict the optimal ordering — using validation performance as the training signal.

Practical Guidelines

• Put the most relevant example last (works for most models).
• Alternate labels to avoid label bias.
• Use consistent formatting across all examples — inconsistency confuses the model.
• Test multiple orderings on a validation set if performance is critical.
• Fix the ordering once determined — don't randomly shuffle at inference time.

Example ordering is an often overlooked but highly impactful aspect of few-shot prompting — the same examples in different orders can produce dramatically different results, making ordering optimization a critical step in prompt engineering.