One Model, Three Modalities: ByteDance Releases Lance for Image and Video Understanding, Generation, and Editing

ByteDance has introduced Lance, a groundbreaking AI model designed to seamlessly integrate the understanding, generation, and editing of both images and videos. Unlike previous approaches that separate these tasks into distinct architectures, Lance uses a single, unified framework trained jointly from the outset. This innovative design allows it to handle a wide array of visual tasks, from captioning and visual question answering to text-to-video generation and intricate editing.

The model's architecture is built on two core principles: unified context modeling and decoupled capability pathways. All inputs—text, images, and videos—are converted into a single, shared multimodal sequence. This allows the model to process diverse data types cohesively. Furthermore, Lance employs a dual-stream mixture-of-experts system to efficiently manage understanding and generation tasks, ensuring that both share context without competing for the same parameters.

Lance demonstrates superior performance across various benchmarks, outperforming many existing unified models. For image generation, it matches top scores on GenEval and performs strongly on DPG-Bench, achieving this with significantly fewer activated parameters than comparable models. In video generation, Lance leads all unified models on VBench, even surpassing several dedicated generation-only models.

To address the complexity of integrating diverse token types within a single sequence, Lance incorporates Modality-Aware Rotary Positional Encoding (MaPE). This mechanism ensures accurate positional encoding for different modalities, preventing ambiguity and improving cross-task alignment. The model undergoes a four-stage training process, including pre-training, continual training, supervised fine-tuning, and reinforcement learning, to refine its capabilities and ensure high accuracy in instruction following and content generation. This comprehensive training regimen, combined with a sophisticated architecture, positions Lance as a significant advancement in multimodal AI.