Connection Pool

Connection Pooling is the technique of maintaining a pre-initialized cache of database connections that are reused across multiple requests — eliminating the expensive per-request overhead of TCP handshake, TLS negotiation, and database authentication that would otherwise dominate latency in high-throughput AI serving applications querying vector databases, relational stores, or caching layers.

What Is Connection Pooling?

• Definition: A managed pool of persistent database connections established at application startup that are borrowed by individual requests, used for their query, then returned to the pool for reuse by the next request — rather than opening and closing a connection for every database interaction.
• The Problem It Solves: Opening a TCP connection involves: DNS resolution (10-50ms), TCP 3-way handshake (1-2 RTT), TLS handshake (1-2 RTT), database authentication (1 RTT) — totaling 50-200ms overhead per connection just to start a query.
• Impact at Scale: An inference server handling 1,000 requests/second without connection pooling would open 1,000 connections per second — exhausting database connection limits and adding 100-200ms of overhead to every single query.
• Pool Economics: A pool of 20 persistent connections can serve thousands of requests per second — each connection handles one query at a time but is immediately available to the next requester.

Why Connection Pooling Matters for AI Systems

• Vector Database Queries: RAG pipelines query vector databases (pgvector, Qdrant, Weaviate, Pinecone) on every user request — pooling eliminates handshake overhead from the critical path of TTFT.
• LLM Caching Layer: Semantic cache lookups in Redis or PostgreSQL happen before every LLM call — pool overhead on these frequent, fast queries would dwarf query execution time.
• Concurrent Inference: 100 concurrent inference requests all need database access simultaneously — a pool of 20 connections queues and serves all 100 without exhausting database limits.
• Metadata Retrieval: Retrieved chunk IDs from vector search must be hydrated with full document metadata from relational DB — a fast, pooled connection makes this hydration sub-millisecond.

Pool Configuration Parameters

Connection Pooling in Python AI Stacks

asyncpg + pgvector (async):
import asyncpg
pool = await asyncpg.create_pool(
dsn="postgresql://user:pass@host/db",
min_size=10, max_size=30
)
async with pool.acquire() as conn:
results = await conn.fetch("SELECT * FROM embeddings WHERE id = $1", chunk_id)

SQLAlchemy (sync/async):
from sqlalchemy.ext.asyncio import create_async_engine
engine = create_async_engine(url, pool_size=20, max_overflow=10)

Redis (aioredis):
import redis.asyncio as aioredis
pool = aioredis.ConnectionPool.from_url("redis://localhost", max_connections=50)
client = aioredis.Redis(connection_pool=pool)

pgBouncer (external proxy):
- Database-side connection pooler for PostgreSQL.
- Multiplexes thousands of application connections through a small pool of real database connections.
- Essential for serverless architectures (Lambda, Modal) where each function invocation creates a new process that would otherwise open its own connection.

Transaction vs Session vs Statement Pooling

Session pooling: One connection per client session — best for stateful operations (transactions, prepared statements). Lowest multiplexing ratio.

Transaction pooling (most common): Connection returned to pool after each transaction. Best for OLTP workloads — connection shared across many clients. Incompatible with prepared statements.

Statement pooling: Connection returned after each statement. Maximum reuse but incompatible with multi-statement transactions.

For AI/RAG workloads: transaction pooling is optimal — queries are short, independent, and high-frequency.

Monitoring Pool Health

Key metrics to track:
- Pool utilization: connections in use / pool max size — alert at > 80%.
- Wait time: time requests spend waiting for available connection — alert at > 10ms.
- Connection errors: failed checkouts due to pool exhaustion — alert any non-zero rate.
- Connection age: maximum connection lifetime to detect stale connections.

Connection pooling is the infrastructure optimization that makes vector database queries invisible in AI serving latency — by eliminating the multi-RTT handshake overhead from every database interaction, connection pooling transforms what would be 100-200ms retrieval bottlenecks into sub-millisecond operations that barely register in the total response time budget.