Researchers Introduce DiscoSign: A Major Breakthrough in Discourse-Aware Sign Language Translation

For years, automated sign language processing systems have faced a fundamental limitation: they have largely operated at the individual sentence level, treating text translation as a series of isolated fragments. While this approach has yielded incremental progress, it inherently ignores the broader context, narrative flow, and critical discourse phenomena that are essential to natural sign language comprehension. Now, a team of researchers—comprising Vasileios Baltatzis, Mert Inan, Connor Gillis, Raja Kushalnagar, Lorna Quandt, Leah Findlater, and Colin Lea—has introduced DiscoSign, a novel computational approach designed to bridge this gap by bringing discourse awareness to text-to-sign language gloss translation.

Grounded deeply in linguistic research, DiscoSign addresses some of the most complex structural challenges inherent in translating spoken or written languages like English into American Sign Language (ASL) glosses. Traditional translation frameworks typically convert text sentence by sentence, which often results in disjointed or grammatically unnatural sign sequences. By developing a modular, Large Language Model (LLM)-based translation framework, the research team has created a system capable of looking beyond the boundaries of a single sentence to maintain coherence across an entire conversation or narrative.

At the core of the DiscoSign framework are solutions to three key linguistic phenomena that have traditionally plagued automated sign language translation systems. The first of these is spatial coreference resolution. In natural sign languages, signers utilize specific locations in the physical signing space to represent entities, characters, or concepts, maintaining consistent spatial references throughout a discourse so that viewers can easily track who or what is being discussed. Conventional sentence-level translation models routinely fail to preserve these spatial assignments from one sentence to the next, leading to spatial confusion. DiscoSign explicitly manages spatial consistency, ensuring that entities maintain their designated spatial locations throughout the entire discourse.

The second major phenomenon addressed by the framework is Question-Answer Clauses, commonly known as QACs. These are pseudocleft structures that serve specific, highly structured discourse functions in sign languages, functioning similarly to rhetorical questions or embedded inquiries that guide the listener’s attention and structure the flow of information. By incorporating mechanisms to handle QACs properly, the DiscoSign framework mirrors the natural communicative patterns found in fluent ASL discourse.

The third pillar of the framework is concept-gloss consistency. Translating between a spoken language and a signed language requires stable and reliable mappings between English concepts and their corresponding ASL signs. Variations in translation can disrupt the fluidity and accuracy of the output, making it difficult for Deaf and Hard-of-Hearing (DHH) individuals to comprehend the generated content smoothly. DiscoSign ensures that these mappings remain stable, reinforcing the reliability of the translation process.

Because traditional evaluation metrics—which were largely designed for spoken-to-spoken language translation or single-sentence text tasks—fail to capture the nuances of discourse-level quality, the researchers also developed a suite of novel evaluation metrics. These metrics are specifically tailored to assess each dimension of discourse coherence addressed by the framework, providing researchers with a rigorous toolset to measure improvements in spatial consistency, clause structure, and overall narrative flow.

Extensive experiments conducted on both sentence-level and discourse-level datasets demonstrated the effectiveness of the new approach. The results reveal that DiscoSign significantly improves spatial consistency and entity tracking compared to traditional sentence-only translation methods. Crucially, these substantial gains in discourse-level coherence are achieved while maintaining competitive performance in single-sentence gloss translation quality, proving that the system does not sacrifice local accuracy for broader structural improvements.

This work establishes the very first systematic framework for discourse-level text-to-sign language gloss translation, complete with a corresponding evaluation methodology that could pave the way for more natural and reliable automated sign language technologies in the future.

Related readings and updates.

AI-driven sign language interpretation has long been constrained by a persistent bottleneck: the scarcity of high-quality annotated data. Although new datasets, including ASL STEM Wiki and FLEURS-ASL, feature professional interpreters and encompass hundreds of hours of video data, they remain only partially annotated. Consequently, these valuable resources are underutilized by the research community, a challenge driven in large part by the prohibitive costs and labor-intensive nature of manual annotation at such a large scale. To address this obstacle, ongoing research efforts include the development of specialized pseudo-annotation pipelines. These systems take signed video and corresponding English text as inputs and automatically output ranked annotations, offering a potential pathway toward more scalable data processing for future sign language models.

At the same time, the broader field of sign language generation continues to evolve to meet the vital needs of the Deaf and Hard-of-Hearing (DHH) community. Sign language generation systems hold immense potential to support seamless communication by translating written languages, such as English, directly into expressive signed videos. However, current generation systems frequently fall short of user expectations due to several persistent shortcomings. These include the poor translation of complex grammatical structures, the complete absence of crucial non-manual markers such as facial expressions and body language, and an overall lack of visual and motion fidelity. Researchers are actively working to address these multifaceted challenges, aiming to build more advanced, expressive, and contextually accurate AI-driven tools that can authentically reflect the richness of natural sign languages.