Self Consistency

Self-Consistency Decoding

What is Self-Consistency?
Self-consistency generates multiple reasoning paths for the same problem, then selects the most common final answer through majority voting.

How It Works

Standard Chain-of-Thought

Problem ---> [Single reasoning path] ---> Answer

Single point of failure: if reasoning is wrong, answer is wrong.

Self-Consistency

Problem ---> [Path 1] ---> Answer A
        ---> [Path 2] ---> Answer B
        ---> [Path 3] ---> Answer A
        ---> [Path 4] ---> Answer A
        ---> [Path 5] ---> Answer C

Majority vote: Answer A (3/5)

Implementation

from collections import Counter

def self_consistent_answer(prompt: str, n_samples: int = 5) -> str:
    answers = []

    for _ in range(n_samples):
        # Sample with temperature > 0 for diversity
        response = llm.generate(
            prompt + "Let us think step by step.",
            temperature=0.7
        )
        answer = extract_final_answer(response)
        answers.append(answer)

    # Majority vote
    counts = Counter(answers)
    return counts.most_common(1)[0][0]

Temperature for Diversity
| Temperature | Effect |
|-------------|--------|
| 0.0 | No diversity, same answer every time |
| 0.5-0.7 | Moderate diversity, good for self-consistency |
| 1.0+ | High diversity, may include wrong paths |

When Self-Consistency Helps

Good Use Cases
| Task | Why It Helps |
|------|--------------|
| Math problems | Multiple valid solution paths |
| Logic puzzles | Different reasoning approaches |
| Code generation | Try multiple implementations |

Less Effective
| Task | Why |
|------|-----|
| Factual recall | Only one correct answer, no reasoning paths |
| Open-ended generation | No "correct" answer to vote on |

Confidence from Agreement
Agreement level indicates confidence:

def get_answer_with_confidence(answers):
    counts = Counter(answers)
    top_answer, top_count = counts.most_common(1)[0]
    confidence = top_count / len(answers)
    return top_answer, confidence

Cost Considerations
| Samples | Accuracy Gain | Cost |
|---------|---------------|------|
| 1 (baseline) | 0% | 1x |
| 3 | ~5-10% | 3x |
| 5 | ~10-15% | 5x |
| 10 | ~15-20% | 10x |

Diminishing returns beyond 5-10 samples.

Self-consistency is especially valuable for high-stakes reasoning where accuracy matters more than cost.