1,686 technical terms and definitions
distributed control system, distributed control, control system
Distributed Control System Introduction & Motivation Distributed Control System addresses a central problem in control-systems process control and control-latency-milliseconds…
pytorch ddp training, gradient synchronization ddp, ddp communication overlap, multi gpu data parallel
Distributed Data Parallel (DDP) is the PyTorch framework for synchronous multi-GPU and multi-node training where each process maintains a full model replica and processes…
Distributed Data Parallel DDP with Ring-AllReduce Introduction & Motivation Distributed Data Parallel DDP with Ring-AllReduce addresses a central problem in optimization algorithm…
design, esd protection network, rail clamp, esd
Electrostatic Discharge (ESD) protection constitutes the dedicated on-chip network of high-current shunting devices engineered to safeguard sensitive gate dielectrics, thin tunnel…
distributed feedback laser mode hop prediction, distributed feedback laser, feedback laser mode, laser mode hop, mode hop prediction, distributed feedback, feedback laser
Distributed Feedback Laser Mode Hop Prediction with Machine Learning Introduction & Motivation Distributed Feedback Laser Mode Hop Prediction with Machine Learning addresses…
lustre, hdfs, parallel file system, gpfs, hpc storage
Distributed and Parallel File Systems (Lustre, HDFS, GPFS) are the storage systems that stripe files across many servers to provide aggregate I/O bandwidth and capacity far beyond…
allreduce gradient synchronization, ring allreduce training, gradient compression communication, parameter server aggregation
Distributed Gradient Aggregation is the process of combining gradient updates computed independently across multiple workers (GPUs or nodes) during distributed deep learning…
gradient quantization, communication reduction training, sparse gradient
Distributed Gradient Compression is the technique of reducing the volume of gradient data communicated between workers during distributed deep learning training, addressing…
consistent hashing distributed, chord kademlia pastry, dht routing performance, peer to peer distributed storage
Distributed Hash Tables (DHT) are decentralized lookup systems that distribute key-value storage across multiple nodes using consistent hashing — providing O(log N) route-to-key…
chord kademlia pastry, peer to peer routing, consistent hashing dht, overlay network routing
Distributed Hash Tables (DHTs) are decentralized lookup systems that provide O(log N) key-value storage and retrieval across N peer nodes without any central coordinator…
dht chord kademlia, consistent hashing partitioning, peer to peer key value store, decentralized lookup routing
Distributed Hash Tables — Decentralized systems that partition key-value storage across multiple nodes, providing scalable lookup operations with guaranteed bounds on routing hops…
model serving distributed, inference parallelism, model sharding serving, inference load balancing
Distributed Inference Serving is the systems engineering discipline of deploying large neural network models across multiple GPUs, multiple machines, or heterogeneous accelerator…
distributed key generation, distributed key, key generation
Distributed Key Generation Introduction & Motivation Distributed Key Generation addresses a central problem in cryptography process control and key-generation-distributed-secret…
distributed hash, consistent hashing, key value database, distributed cache
Distributed Key-Value Stores are systems that partition key-value data across multiple nodes using consistent hashing or range partitioning — providing horizontal scalability…
distributed hash table, consistent hashing, partitioned database
Distributed Key-Value Stores are systems that partition a key-value dataset across multiple nodes, providing scalable storage and retrieval with tunable consistency, availability…
distributed ledger consensus protocol prediction, distributed ledger consensus, ledger consensus protocol, consensus protocol prediction, distributed ledger
Distributed Ledger Consensus Protocol Prediction with Machine Learning Introduction & Motivation Distributed Ledger Consensus Protocol Prediction with Machine Learning addresses…
distributed lock management, distributed lock, lock management
Distributed Lock Management Introduction & Motivation Distributed Lock Management addresses a central problem in distributed-systems process control and lock-acquisition-latency…
message passing model, halo exchange, ghost cells, parallel domain decomp, mpi domain decomposition
Distributed Memory Programming and Domain Decomposition is the parallel computing methodology where a large computational domain is partitioned into subdomains, each processed…
designing distributed power delivery networks for advanced nodes, designing distributed power, distributed power delivery, power delivery networks, delivery networks advanced
Designing distributed power delivery networks for advanced nodes Introduction & Motivation Designing distributed power delivery networks for advanced nodes addresses a central…
distributed real-time synchronization, distributed real-time, real-time synchronization
Distributed Real-Time Synchronization Introduction & Motivation Distributed Real-Time Synchronization addresses a central problem in real-time process control…
rag
Distributed retrieval is the retrieval architecture that partitions indexes and query execution across multiple nodes or regions - it enables high availability and large-scale…
dsm, software dsm, partitioned global address, pgas
Distributed Shared Memory (DSM) and PGAS are the programming abstractions that present a single shared address space to processes running on physically separate machines, each…
dsm systems design, virtual shared address space, memory coherence distributed, scalable shared memory
Distributed Shared Memory Architecture — DSM systems provide a shared memory abstraction over physically distributed memory nodes, enabling transparent data access across…
memory consistency model, coherence protocol, dsm system
Distributed Shared Memory (DSM) and Consistency Models define how memory operations across multiple processors are ordered and made visible to other processors, establishing…
Distributed Consensus, Paxos, Raft, and Byzantine Replication Distributed consensus allows independent machines to behave like one reliable state machine despite delay…
distributed computing, cap theorem, consensus protocol, fault tolerance, distributed training
Distributed systems is the engineering discipline of building reliable, scalable computing services across multiple networked machines that coordinate to appear as a single…
mlops
Distributed tracing is an observability technique that tracks a single request as it flows through multiple services in a distributed system, recording timing, metadata…
distributed tracing correlation, distributed tracing, tracing correlation
Distributed Tracing Correlation Introduction & Motivation Distributed Tracing Correlation addresses a central problem in ml-ops process control…
distributed tracing latency, distributed tracing, tracing latency
Distributed Tracing Latency Introduction & Motivation Distributed Tracing Latency addresses a central problem in cloud process control and trace-sampling-rate-accuracy prediction…
ddp, fsdp
Distributed Training Training Paradigms Data Parallel (DDP) Each GPU has full model copy, processes different data: Model Parallel Split model across GPUs: Tensor Parallel: Split…
data parallel training, ddp, fsdp, tensor parallel, pipeline parallel, expert parallel, nccl
Distributed training coordinates optimization across multiple accelerators and often multiple nodes. Large language, vision, recommendation, scientific, and multimodal models…
data parallel training pytorch, ddp distributed data parallel, gradient synchronization training, data parallel scaling efficiency
Data Parallel Distributed Training is the most widely used strategy for scaling deep learning training across multiple GPUs or nodes by replicating the entire model on each…
horovod distributed, pytorch distributed, deepspeed training, distributed ml framework
Distributed Training Frameworks are the software systems that coordinate the training of large machine learning models across multiple GPUs and multiple machines — handling data…
hierarchical all-reduce algorithm, multi-level allreduce
Hierarchical all-reduce is the two-level collective strategy that reduces gradients within nodes first, then across nodes - it exploits faster intra-node links and minimizes…
distributed training scalability and communication, distributed training scalability, training scalability communication, distributed training, training scalability
Distributed Training Scalability and Communication Introduction & Motivation Distributed Training Scalability and Communication addresses a central problem in ml-training process…
weak strong scaling analysis, communication overhead scaling, parallel efficiency metrics, scalability bottlenecks
Distributed Training Scaling Efficiency is the measure of how effectively training performance improves with additional compute resources — quantified through strong scaling…
distributed training synchronization, distributed training, training synchronization
Distributed Training Synchronization Introduction & Motivation Distributed Training Synchronization addresses a central problem in systems process control…
distributed transaction commit, distributed transaction, transaction commit
Distributed Transaction Commit Introduction & Motivation Distributed Transaction Commit addresses a central problem in distributed-systems process control…
semi-supervised learning
Distribution Alignment is a technique in semi-supervised learning that adjusts pseudo-label distributions to match the true class distribution — preventing the model from being…
robustness
Distribution Shift is the discrepancy between the data distribution during training and the distribution encountered during deployment — when $P_{test}(X, Y) eq P_{train}(X, Y)$…
ai safety
Distribution Shift is the change between training-time data distribution and real-world deployment data over time or context - It is a core method in modern AI safety execution…
reinforcement learning advanced
Distributional Bellman is Bellman operators over full return distributions instead of only expected scalar value. - It models uncertainty and multimodal outcomes that…
Distributional Reinforcement Learning (C51) Introduction & Motivation Distributional Reinforcement Learning (C51) addresses a central problem in sequential decision making via…
Distributional Reinforcement Learning (C51) (Extended Study) Introduction & Motivation Distributional Reinforcement Learning (C51) (Extended Study) addresses a central problem…
Distributional Reinforcement Learning (C51) (Extended Study) Introduction & Motivation Distributional Reinforcement Learning (C51) (Extended Study) addresses a central problem…
Distributional Reinforcement Learning Introduction & Motivation Distributional Reinforcement Learning addresses a central problem in reinforcement policy learning…
reinforcement learning
Distributional RL is a reinforcement learning framework that learns the full distribution of returns (not just the expected value) — instead of estimating , distributional RL…
Distributional RL and Quantile Regression Introduction & Motivation Distributional RL and Quantile Regression addresses a central problem in reinforcement policy learning…
Distributional Robustness Optimization Introduction & Motivation Distributional Robustness Optimization addresses a central problem in uncertainty methods and prediction…
distributionally robust optimization (dro) prediction, distributionally robust optimization, robust optimization (dro), optimization (dro) prediction, robust optimization
Distributionally Robust Optimization (DRO) Prediction with Machine Learning Introduction & Motivation Distributionally Robust Optimization (DRO) Prediction with Machine Learning…