Weight Entanglement

Weight Entanglement is a phenomenon in weight-sharing NAS methods where the shared weights of sub-networks interfere with each other — preventing accurate performance estimation because training one sub-network path affects the weights used by other paths.

What Is Weight Entanglement?

• Problem: In one-shot NAS (like DARTS), all sub-networks share the same set of weights. Training improves one sub-network but may degrade others.
• Consequence: The ranking of sub-architectures using shared weights does not match their ranking when trained independently.
• Severity: More severe with larger search spaces and more shared paths.

Why It Matters

• NAS Reliability: Weight entanglement is the primary reason one-shot NAS methods sometimes find sub-optimal architectures.
• Solutions: Progressive shrinking (OFA), few-shot NAS (split into multiple sub-supernets), or training longer to reduce interference.
• Research: Understanding and mitigating weight entanglement is an active area of NAS research.

Weight Entanglement is the interference pattern in shared-weight NAS — where training one architecture pathway inadvertently disrupts the performance of other pathways.