Tensor Fusion

Tensor Fusion is a multimodal fusion technique that captures all possible cross-modal interactions by computing the outer product of modality-specific feature vectors — creating a high-dimensional tensor that explicitly encodes unimodal, bimodal, and trimodal feature interactions, enabling the model to discover complex inter-modal correlations that simpler fusion methods miss.

What Is Tensor Fusion?

• Definition: Given feature vectors from N modalities, tensor fusion computes their outer product to create an N-dimensional tensor containing every possible feature interaction across modalities.
• Outer Product: For vision V ∈ R^v, audio A ∈ R^a, and language L ∈ R^l, the fused tensor T = V ⊗ A ⊗ L ∈ R^(v×a×l) captures all v·a·l cross-modal interactions.
• Augmented Vectors: Each modality vector is augmented with a constant 1 (e.g., V' = [V; 1]) before the outer product, ensuring the tensor also contains unimodal and bimodal terms alongside trimodal interactions.
• Tensor Fusion Network (TFN): The original architecture by Zadeh et al. (2017) that introduced this approach for multimodal sentiment analysis, achieving state-of-the-art results on CMU-MOSI and IEMOCAP benchmarks.

Why Tensor Fusion Matters

• Complete Interaction Modeling: Unlike concatenation (which only captures unimodal features) or bilinear fusion (which captures pairwise interactions), tensor fusion explicitly models all orders of cross-modal interaction in a single representation.
• Expressiveness: The outer product creates a feature space rich enough to represent subtle correlations — such as how a specific facial expression combined with a particular tone of voice and specific word choice indicates sarcasm.
• Theoretical Foundation: Tensor fusion provides a mathematically principled way to combine modalities, with connections to polynomial feature expansion and kernel methods.
• Benchmark Performance: TFN achieved significant improvements on multimodal sentiment analysis, emotion recognition, and speaker trait recognition tasks.

Scalability Challenge and Solutions

• Dimensionality Explosion: The outer product of three 256-dimensional vectors produces a 256³ ≈ 16.7 million dimensional tensor — computationally prohibitive for large feature dimensions.
• Low-Rank Approximation (LMF): Decomposes the full tensor into a sum of R rank-1 tensors, reducing complexity from O(d^N) to O(R·N·d) while preserving most interaction information.
• Factorized Multimodal Transformer: Uses attention mechanisms to implicitly compute tensor interactions without materializing the full tensor.
• Tucker Decomposition: Represents the interaction tensor as a core tensor multiplied by factor matrices, providing a tunable compression ratio.

Tensor fusion provides the most complete multimodal interaction modeling — computing outer products across modality features to capture every possible cross-modal correlation, with low-rank approximations making this powerful approach practical for real-world multimodal AI systems.