Thread

Threads vs Processes in Python AI is the critical concurrency architecture decision governed by Python's Global Interpreter Lock (GIL) — threads are correct for I/O-bound LLM API calls and database queries, while multiprocessing is necessary for CPU-bound operations like tokenization, preprocessing, and data augmentation that need true parallelism.

What Is the Python GIL?

• Definition: The Global Interpreter Lock is a mutex in CPython that prevents multiple threads from executing Python bytecode simultaneously — only one thread can hold the GIL and run Python code at any moment, even on a multi-core CPU.
• Why It Exists: Python's memory management (reference counting) is not thread-safe. The GIL prevents race conditions in the interpreter's internal state without requiring fine-grained locking on every object.
• Critical Impact: A 32-core server running a pure Python computation in 8 threads achieves no speedup vs 1 thread — all 8 threads time-share a single core's worth of Python execution due to the GIL.
• When GIL Is Released: The GIL IS released during I/O operations (network, disk), C extension code (NumPy, PyTorch CUDA kernels), and time.sleep() — enabling genuine concurrency for I/O-bound tasks.

Threads: When to Use

Python threads ARE effective when:
- Network I/O: Calling OpenAI API, fetching from vector DB, querying Redis. Thread releases GIL while waiting for network — other threads run.
- File I/O: Reading documents from disk, writing logs. GIL released during kernel I/O.
- C Extension Work: NumPy and PyTorch operations release the GIL — multiple threads can run CPU-bound NumPy array operations in parallel.
- Wait-heavy workloads: Threads holding the GIL for microseconds between I/O calls — overhead is negligible.

from concurrent.futures import ThreadPoolExecutor
import httpx

def call_api(prompt: str) -> str:
# Network I/O — GIL released while waiting
return httpx.post(LLM_API_URL, json={"prompt": prompt}).json()

with ThreadPoolExecutor(max_workers=20) as executor:
results = list(executor.map(call_api, prompts)) # 20 concurrent API calls

Processes: When to Use

Python multiprocessing IS required when:
- CPU-bound preprocessing: Tokenization, text cleaning, BPE encoding, audio transcoding — pure Python CPU work not releasing GIL.
- Data augmentation: Image transforms (resize, crop, normalize) in pure Python — needs real parallelism.
- Parallelizing training loops: Each process gets its own Python interpreter and GIL — true parallel execution.
- DataLoader workers: PyTorch DataLoader uses multiprocessing for workers — each worker process independently loads and preprocesses batches.

from torch.utils.data import DataLoader
# num_workers > 0 → multiprocessing, each worker is a separate process
dataloader = DataLoader(dataset, batch_size=32, num_workers=8)

from multiprocessing import Pool
def tokenize_document(doc: str) -> list[int]:
return tokenizer.encode(doc) # CPU-bound — needs true parallelism

with Pool(processes=8) as pool:
token_lists = pool.map(tokenize_document, documents)

The Correct Concurrency Model for AI Systems

Memory Model Differences

Threads: Shared memory space — all threads see the same Python objects. Fast to create (~microseconds), low memory overhead, but requires locks for shared mutable state.

Processes: Separate memory spaces — each process has its own copy of all data. Slow to create (fork: ~milliseconds), high memory overhead (copy-on-write until modified), but completely isolated — crashes do not propagate.

IPC (Inter-Process Communication):
- Queue/Pipe: Pass data between processes via serialization (pickle).
- Shared Memory (multiprocessing.shared_memory): Zero-copy sharing of arrays between processes.
- Memory-mapped files: Share large datasets across processes.

GIL in Python 3.13+

Python 3.13 introduces optional free-threading (GIL-free) mode — early support, not yet production-ready for most AI workloads. The GIL remains the default. This will eventually change the threads-vs-processes calculus for CPU-bound Python code, but for now the rules above apply.

The threads vs processes decision is the architectural foundation of Python AI system performance — choosing threads for I/O-bound LLM API calls enables efficient concurrency, while choosing multiprocessing for CPU-bound preprocessing enables the true parallelism that multi-core hardware provides, together ensuring that neither the network nor the CPU becomes an unnecessary bottleneck.