Researchers Unveil First Tri-Modal Masked Diffusion Models Pretrained from Scratch on Text, Image, and Audio Data

In a significant milestone for artificial intelligence and generative modeling, a large team of researchers has introduced the first tri-modal Masked Diffusion Models (MDM) pretrained entirely from scratch on a diverse mixture of text, image-text, and audio-text data. Authored by Louis Béthune, Victor Turrisi, Bruno Kacper Mlodozeniec, Pau Rodriguez Lopez, Lokesh Boominathan, Nikhil Bhendawade, Amitis Shidani, Joris Pelemans, Theo X. Olausson, Devon Hjelm, Paul Dixon, João Monteiro, Pierre Ablin, Vishnu Banna, Arno Blaas, Nick Henderson, Kari Noriy, Dan Busbridge, Josh Susskind, Marco Cuturi, Irina Belousova, Luca Zappella, Russ Webb, and Jason Ramapuram, the new study marks a major departure from traditional methodologies that rely heavily on autoregressive language models.

Discrete diffusion models have increasingly emerged as robust, highly competitive alternatives to conventional autoregressive architectures. Until now, however, prevailing approaches in the field typically relied on initializing and subsequently fine-tuning a base unimodal model to achieve bi-modal generation capabilities. Diverging sharply from these earlier pathways, the research team has successfully constructed and pretrained a tri-modal architecture from the ground up. By incorporating text, image-text, and audio-text data simultaneously into the foundational pretraining phase, the investigators have opened up new possibilities for unified, multi-sensory machine learning systems capable of handling multiple data modalities natively within a single framework.

To understand the mechanics and optimal operating conditions of this massive architecture, the research team conducted a rigorous, systematic analysis of multimodal scaling laws. Their investigation went far beyond simple model size expansion, probing deeply into the complex interplay of various training dynamics. Specifically, the scientists analyzed modality mixing ratios, noise schedules, batch-size effects, and derived optimized inference sampling defaults. These foundational insights are critical for guiding future developments in large-scale discrete diffusion research, ensuring that models can be scaled efficiently without encountering unexpected bottlenecks or performance degradations.

Among the technical breakthroughs highlighted in the study, the team’s batch-size analysis yielded a novel stochastic differential equation (SDE) based reparameterization. In recent machine learning literature, tuning the optimal batch size has often presented a persistent challenge, requiring delicate adjustments to balance computational efficiency and training stability. The newly introduced SDE-based reparameterization successfully eliminates the need for manual batch-size tuning. By mathematically separating physical and logical considerations, the reparameterization effectively decouples the physical batch size—which is typically chosen based on hardware constraints such as GPU saturation, floating-point operation (FLOP) efficiency, and wall-clock time—from the logical batch size, which is selected specifically to balance the variance of gradients during stochastic optimization. This separation provides engineers and researchers with unprecedented flexibility when deploying large-scale training runs across diverse hardware clusters.

To demonstrate the practical viability and raw power of their unified design philosophy, the research team pretrained a preliminary large-scale model. Standing at the impressive three-billion-parameter scale and trained on a staggering 6.4 trillion tokens, the preliminary model achieved strong, highly competitive results across multiple domains. These include traditional text generation tasks, text-to-image (T2I) generation, and text-to-speech (T2S) tasks. By demonstrating high-fidelity outputs across text, visual, and auditory modalities from a single cohesive architecture, the work underscores the viability of discrete diffusion as a universal backbone for generative AI applications.

This comprehensive investigation represents the largest-scale systematic open study of multimodal discrete diffusion models conducted to date. By releasing their findings and insights into scaling behaviors across multiple modalities, the authors have provided the broader scientific and engineering community with a valuable roadmap for future research and development in unified generative frameworks.

Related readings and updates.

Discrete diffusion remains a highly promising framework for modeling and generating complex discrete data distributions. Expanding upon the core principles of discrete diffusion, related ongoing research continues to refine the underlying mathematical objectives and training frameworks. Among these advancements is Target Concrete Score Matching (TCSM), introduced as a novel and versatile objective specifically designed for training and fine-tuning discrete diffusion models. TCSM provides a flexible framework with remarkably broad applicability across various machine learning tasks. It directly supports the pretraining of discrete diffusion models straight from raw data samples, and many existing discrete diffusion approaches naturally emerge as special cases or extensions of this holistic framework.

Parallel to developments in scoring and objective functions, maintaining stable training dynamics across varying batch sizes remains an essential tool for practical, large-scale machine learning engineering. Effective scaling rules allow researchers to navigate the trade-off between physical batch size and wall-clock training time efficiently. For instance, standard methodologies in stochastic gradient descent dictate scaling the learning rate linearly with the batch size to preserve trajectory dynamics. Similarly, model Exponential Moving Average (EMA), which maintains a functional, smoothed copy of a target model throughout training, plays a critical role in stabilizing final performance. Understanding how to properly scale these mechanisms alongside modern discrete diffusion architectures ensures that practitioners can maximize both computational throughput and model accuracy in future iterations of large-scale multimodal systems.