Distributed Consensus Advanced
Advanced Distributed Consensus covers optimized implementations and variants of consensus protocols (Raft, Multi-Paxos, and beyond) that achieve high throughput and low latency for replicated state machines in production distributed systems — going beyond basic correctness to address real-world performance, batching, pipelining, and multi-group scalability.
Multi-Paxos Optimizations: Basic Paxos requires 2 round-trips per value (Prepare + Accept). Multi-Paxos elects a stable leader that skips the Prepare phase for subsequent proposals, achieving single round-trip (Accept only) in steady state. Further optimizations:
- Batching: Group multiple client requests into a single consensus round. A leader accumulates requests for a configurable window (e.g., 1ms or 100 requests) then proposes the batch as one log entry. Amortizes consensus overhead across many operations, increasing throughput 10-100x while adding small latency.
- Pipelining: Don't wait for one proposal to complete before starting the next. The leader issues proposals for log slots n, n+1, n+2... concurrently, with each Accept round in flight simultaneously. Increases throughput by overlapping consensus rounds with network latency.
- Parallel commit: For commutative operations, multiple log positions can commit independently without total ordering. Generalized Paxos and EPaxos exploit this for higher throughput on non-conflicting operations.
Raft Optimizations:
| Optimization | Mechanism | Benefit |
|---|---|---|
| Log batching | Bundle entries per AppendEntries | Higher throughput |
| Pipelining | Send next batch before ack | Hide network latency |
| Read leases | Leader serves reads without consensus | 10-100x read throughput |
| Pre-vote | Check electability before election | Avoid disruptive elections |
| Joint consensus | Two-phase membership change | Safe reconfiguration |
| Learner nodes | Non-voting replicas | Read scaling |
Read Optimizations: Linearizable reads typically require a consensus round (to confirm leadership is current). Alternatives: ReadIndex — leader confirms majority heartbeat, then serves read from current commit (one round trip, no log entry); Lease-based reads — leader holds a time-based lease during which it's guaranteed to be leader, serving reads locally with no communication (requires synchronized clocks within lease duration).
Multi-Group Consensus: A single Raft/Paxos group becomes a bottleneck at high throughput. Production systems shard data across many consensus groups (e.g., TiKV uses one Raft group per data region, CockroachDB per range). Challenges: colocated groups — a single server participates in thousands of Raft groups, requiring efficient multiplexing of heartbeats and log storage; cross-group transactions — operations spanning multiple groups require two-phase commit layered above consensus; and group management — splitting, merging, and rebalancing groups as data grows.
State Machine Replication Pitfalls: Log compaction — unbounded log growth requires periodic snapshotting; snapshot transfer to slow followers must not block normal operation. Membership changes — adding/removing nodes safely requires consensus protocol support to avoid split-brain. Disk I/O — consensus requires durable writes (fsync) on the critical path; batching fsync operations is essential for performance.
Advanced consensus protocols achieve the seemingly impossible — strong consistency with throughputs of millions of operations per second and sub-millisecond latency — through careful engineering of batching, pipelining, and read optimizations that reduce the cost of agreement to its theoretical minimum.