DiscoSign: A New Framework for Discourse-Aware Sign Language Translation

Sign Language Translation Has Long Been Stuck at the Sentence Level

Mainstream sign language processing systems have a largely overlooked weakness: they almost universally process input one sentence at a time, with no ability to capture cross-sentence discourse phenomena. Yet a genuine understanding of sign language depends critically on exactly this kind of discourse-level information. The DiscoSign framework, proposed by a team of researchers, is designed to close this gap — it is a discourse-aware text-to-sign language gloss translation approach grounded in linguistic research.

In other words, previous systems translated each sentence into sign glosses in isolation, much like translating a conversation while ignoring who "he" refers to in context. For a language like sign language — which relies heavily on spatial structure and discourse organization — this isolated processing leads to significant information loss.

Three Discourse Phenomena DiscoSign Addresses

DiscoSign is built on a modular large language model (LLM) translation framework, with a focus on three key discourse phenomena.

Spatial Coreference Resolution

The first is spatial coreference resolution. In sign languages, once an entity is introduced, it is typically assigned a fixed location in the signing space in front of the signer, and subsequent references to that entity are made by pointing to that location. This means the same entity must maintain consistent spatial placement throughout an entire discourse. Sentence-level systems cannot preserve this cross-sentence spatial consistency, and DiscoSign explicitly models this during translation.

Question-Answer Clauses

The second phenomenon is Question-Answer Clauses (QACs) — a type of pseudocleft structure that plays a key role in organizing information and marking focus in sign languages. Rather than conveying information through a simple declarative statement, QACs use a "pose a question, then answer it" format, making them a common and important rhetorical and structural device in sign language discourse. Correctly identifying and generating these structures is critical for the naturalness and comprehensibility of translations.

The Third Phenomenon and Modular Design

The framework also addresses a third class of discourse phenomena (truncated in the available abstract). Overall, the core idea behind DiscoSign is to translate these well-studied linguistic phenomena into modular, processable components within an LLM-based translation pipeline — enabling the system to account for whole-discourse coherence rather than focusing narrowly on individual sentences.

Why This Work Matters

Sign language is not simply a word-for-word rendering of spoken language — it is a complete natural language with its own grammar, spatial semantics, and discourse rules. Previous technical approaches have often overlooked this, resulting in gloss sequences that are locally correct sentence by sentence but globally incoherent and lacking discourse cues.

The value of DiscoSign lies in combining linguistic research findings with the capabilities of large language models: leveraging the LLM's powerful text understanding and generation abilities on one hand, and on the other, using a modular design to explicitly inject sign-language-specific discourse rules. This "linguistic knowledge + large model" integration approach has real-world significance for improving the usability of sign language translation and, by extension, the information accessibility of the deaf and hard-of-hearing community.

For the research community, this work also points to a more general direction: in low-resource, structurally distinctive language processing tasks, purely data-driven end-to-end models may not be optimal. Introducing domain-specific linguistic knowledge through modular components may yield more reliable and interpretable results.

Summary

DiscoSign advances sign language translation from isolated sentence-level processing to the discourse level, focusing on key phenomena such as spatial coreference resolution and Question-Answer Clauses, implemented through a modular LLM framework. While the publicly available abstract is limited in detail, the research direction clearly addresses a long-underappreciated problem in sign language processing. As more details and experimental results are released, discourse-aware approaches like this one have the potential to become an important baseline for sign language machine translation.