Mpi Derived Datatypes
MPI Derived Datatypes are user-defined type descriptors that enable efficient communication of noncontiguous, heterogeneous, or structured data without manual packing into contiguous buffers — allowing MPI to directly access scattered memory locations during send/receive operations with optimal zero-copy performance on supported networks.
Type Constructor Hierarchy:
- MPI_Type_contiguous: creates a type from N consecutive copies of an existing type — simplest constructor, equivalent to a C array
- MPI_Type_vector/hvector: describes N blocks of count elements with fixed stride between blocks — ideal for matrix columns, subarray slices, and strided grid data; hvector specifies stride in bytes for heterogeneous layouts
- MPI_Type_indexed/hindexed: each block has individually specified offset and size — handles irregular access patterns like sparse matrix rows or adaptive mesh element lists
- MPI_Type_create_struct: most general constructor combining different base types at arbitrary byte offsets — maps directly to C structs with mixed types and padding
Zero-Copy Protocol:
- Packing Avoidance: when hardware supports scatter-gather (InfiniBand, Omni-Path), derived datatypes enable direct RDMA from noncontiguous memory without copying to intermediate buffers — eliminating the serialization overhead of MPI_Pack/MPI_Unpack
- Type Commit Optimization: MPI_Type_commit analyzes the type map and selects the optimal data access strategy — pipelining scattered reads with network transfers for large messages
- Dataloop Representation: internal representation of committed types as iteration patterns (loops over blocks with stride/offset) enables efficient traversal without per-element function calls
- Network Offload: modern interconnects (UCX, libfabric) can offload derived datatype processing to the NIC for hardware-accelerated scatter-gather DMA
Common Patterns:
- Matrix Subarray: MPI_Type_create_subarray extracts an N-dimensional subblock from a larger array — used for halo exchange in structured grid codes, distributing 2D/3D domain decompositions
- Struct Serialization: defining MPI types matching C/Fortran structs enables direct communication of record-oriented data without manual field-by-field packing
- Indexed Scatter: MPI_Type_indexed with per-element offsets enables gather/scatter patterns — extracting boundary nodes from unstructured mesh data or communicating sparse vector entries
Performance Considerations:
- Small Message Overhead: for very small messages (<1 KB), the overhead of type traversal may exceed manual packing cost — benchmark before adopting derived types for latency-sensitive small messages
- Nested Type Depth: deeply nested type constructors (types built from types built from types) can cause performance degradation in some MPI implementations — flattening to indexed types may help
- Memory Registration: RDMA-based transports require memory registration for zero-copy; scattered pages may require multiple registrations, partially negating the benefit of avoiding packing
MPI derived datatypes are an essential abstraction for scientific computing that eliminates error-prone manual data serialization while enabling MPI implementations to optimize noncontiguous data transfer — achieving both programmer productivity and communication performance for complex distributed data structures.