Low-Rank Tensor Fusion

Low-Rank Tensor Fusion (LMF) is an efficient multimodal fusion method that approximates the full tensor outer product using low-rank decomposition — reducing the computational complexity of tensor fusion from exponential to linear in the number of modalities while preserving the ability to model cross-modal interactions, making expressive multimodal fusion practical for real-time applications.

What Is Low-Rank Tensor Fusion?

• Definition: LMF approximates the weight tensor W of a multimodal fusion layer as a sum of R rank-1 tensors, where each rank-1 tensor is the outer product of modality-specific factor vectors, avoiding explicit computation of the full high-dimensional tensor.
• Decomposition: W ≈ Σ_{r=1}^{R} w_r^(1) ⊗ w_r^(2) ⊗ ... ⊗ w_r^(M), where w_r^(m) are learned factor vectors for each modality m and rank component r.
• Efficient Computation: Instead of computing the d₁×d₂×d₃ tensor explicitly, LMF computes R inner products per modality and combines them, reducing complexity from O(∏d_m) to O(R·Σd_m).
• Origin: Proposed by Liu et al. (2018) as a direct improvement over the Tensor Fusion Network, achieving comparable accuracy with orders of magnitude fewer parameters.

Why Low-Rank Tensor Fusion Matters

• Scalability: Full tensor fusion on three 256-dim modalities requires ~16.7M parameters; LMF with rank R=4 requires only ~3K parameters — a 5000× reduction enabling deployment on mobile and edge devices.
• Speed: Linear complexity in feature dimensions means LMF runs in milliseconds even for high-dimensional modality features, enabling real-time multimodal inference.
• Preserved Expressiveness: Despite the dramatic parameter reduction, LMF retains the ability to model cross-modal interactions because the low-rank factors span the most important interaction subspace.
• End-to-End Training: All factor vectors are jointly learned through backpropagation, automatically discovering the most informative cross-modal interaction patterns.

How LMF Works

• Step 1 — Modality Encoding: Each modality is encoded into a feature vector by its respective sub-network (CNN for images, LSTM/Transformer for text, spectrogram encoder for audio).
• Step 2 — Factor Projection: Each modality feature is projected through R learned factor vectors, producing R scalar values per modality.
• Step 3 — Rank-1 Combination: For each rank component r, the scalar projections from all modalities are multiplied together, capturing the cross-modal interaction for that component.
• Step 4 — Summation: The R rank-1 interaction values are summed and passed through a final classifier layer.

Low-rank tensor fusion makes expressive multimodal interaction modeling practical — decomposing the prohibitively large tensor outer product into a compact sum of rank-1 components that preserve cross-modal correlation capture while reducing parameters by orders of magnitude, enabling real-time multimodal AI on resource-constrained platforms.