Researchers Introduce First Tri-Modal Masked Diffusion Models Scaling to 3 Billion Parameters

The landscape of generative artificial intelligence is undergoing a significant architectural evolution as researchers continue to push beyond the boundaries of traditional autoregressive language models. In a notable development for the field, a large team of researchers—including Louis Béthune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, and numerous other contributors from across the artificial intelligence research community—has introduced the first tri-modal Masked Diffusion Models (MDM) pretrained completely from scratch on a diverse mixture of text, image-text, and audio-text data.

Discrete diffusion models have increasingly emerged within the academic and industrial research spheres as powerful alternatives to the dominant autoregressive paradigm. While prior methodologies have predominantly relied on initializing and subsequently fine-tuning a base unimodal model to handle bi-modal generation tasks, this latest work breaks new ground by departing entirely from previous approaches. By building a tri-modal framework from the ground up, the research team has opened new pathways for true cross-modal understanding and generation, handling text, images, and audio natively within a single, unified architectural design.

The release of this comprehensive study marks the largest-scale open investigation of multimodal discrete diffusion models conducted to date. To achieve this milestone, the researchers undertook a systematic and rigorous analysis of several critical training dynamics and architectural configurations. Their investigation deeply explored multimodal scaling laws, optimal modality mixing ratios, complex noise schedules, and the subtle yet powerful effects of batch-size variations on overall model convergence. Furthermore, the team formulated and provided highly optimized inference sampling defaults, ensuring that the theoretical advances made during the research phase can be practically harnessed by other practitioners and researchers in the broader artificial intelligence ecosystem.

Among the core technical hurdles addressed in the study is the complex relationship between physical and logical batch sizes during stochastic optimization. Traditionally, scaling models of this magnitude requires careful calibration of batch sizes to balance hardware constraints—such as GPU memory saturation, floating-point operations per second (FLOP) efficiency, and wall-clock training time—against the statistical necessity of maintaining low variance in gradient updates. To solve this dilemma, the authors’ batch-size analysis yielded a novel stochastic differential equation (SDE) based reparameterization.

This innovative reparameterization effectively eliminates the cumbersome requirement of tuning the optimal batch size, a persistent bottleneck reported in recent generative modeling literature. By successfully decoupling the physical batch size, which is dictated by computational infrastructure, from the logical batch size, which is chosen to optimize gradient variance, the new SDE-based framework streamlines the training of massive multimodal architectures. This ensures that training dynamics can be rigorously preserved across varying hardware setups, offering a vital tool for practical machine learning engineering where balancing wall-clock time and computational resources is paramount.

To demonstrate the viability and raw power of their unified design, the research team successfully pretrained a preliminary model at the massive scale of 3 billion parameters, consuming a staggering 6.4 trillion tokens during the training process. Despite the inherent difficulties of training discrete diffusion models across multiple distinct data modalities simultaneously, this large-scale model achieved remarkably strong empirical results. The preliminary 3B model demonstrated robust performance across a diverse suite of benchmarks, excelling not only in traditional text generation tasks but also showcasing advanced capabilities in text-to-image (T2I) generation and text-to-speech (T2S) synthesis.

By proving that a single discrete diffusion architecture can simultaneously master text, visual, and auditory domains at a multi-billion parameter scale, this work provides the broader scientific community with invaluable insights into the scaling behaviors of non-autoregressive multimodal systems. The findings suggest that discrete diffusion may soon offer a highly competitive, flexible alternative to autoregressive transformers for foundational, multi-sensory artificial intelligence models.

Related readings and updates

As the field of discrete diffusion continues to mature, researchers are actively developing foundational frameworks and training objectives to make these models more versatile and stable. Discrete diffusion remains a deeply promising framework for modeling and generating complex discrete data distributions. Among the recent advancements complementing these large-scale studies is Target Concrete Score Matching, abbreviated as TCSM.

TCSM has been introduced as a novel and highly versatile objective specifically designed for training and fine-tuning discrete diffusion models. Offering a generalized framework with broad applicability across different network architectures, TCSM directly supports the pre-training of discrete diffusion models straight from raw data samples. Notably, many existing discrete diffusion approaches naturally emerge as special cases within this broader theoretical formulation.

Alongside architectural objectives, maintaining stable training dynamics across varying hardware configurations remains an essential focus for practical machine learning development. Practical tools such as scaling rules—for instance, scaling the learning rate linearly with the batch size in stochastic gradient descent—allow engineers to effectively trade off batch size against wall-clock training time. Another crucial component in this optimization toolkit is the exponential moving average (EMA) of model weights. Serving as a functional copy of a target neural network, proper tuning and scaling of the EMA mechanism ensures that researchers can reliably capture optimal model checkpoints during the intensive pre-training phases required for large-scale multimodal deployments.