Sundae

SunDAE (Step-unrolled Denoising Autoencoder) is a non-autoregressive text generation model that iteratively denoises a corrupted sequence — starting from a randomly corrupted input and applying a denoising model repeatedly, with the key innovation of unrolling denoising steps during training for improved multi-step generation.

SunDAE Approach

• Corruption: Randomly corrupt the target sequence — random token replacement, masking, or insertion.
• Denoising: Train a model to reconstruct the clean sequence from the corrupted version.
• Unrolled Training: During training, perform multiple denoising steps and backpropagate through all steps — trains for iterative refinement.
• Generation: At inference, start from random tokens and iteratively denoise — each step improves the output.

Why It Matters

• Training-Inference Alignment: Unrolled training aligns the training objective with the iterative inference procedure — reduces the train-test gap.
• Simple: No complex scheduling or masking strategy needed — just corrupt, denoise, and repeat.
• Competitive: Achieves competitive performance with more complex non-autoregressive methods on machine translation.

SunDAE is iterative denoising with training that matches inference — unrolling denoising steps during training for better non-autoregressive text generation.