SignTrace: Describe a Sign in Natural Language, Find the Sign Language Word

Sign language learners often run into a frustrating problem: they clearly remember the motion of a particular sign, but have no idea what it means or how to encode it into a formal query for a dictionary. This "reverse lookup" challenge has long troubled beginners and researchers alike. A new system called SignTrace aims to solve this pain point through natural language — just describe a sign's movement in everyday words, and the system will find the corresponding vocabulary entry and its meaning.

Reverse Lookup: The Overlooked Challenge in Sign Language Search

Traditional dictionaries work in one direction: you know a word and look up its meaning. But sign language learning presents a unique challenge — learners often remember the physical motion of a sign before knowing which word it represents or what it means.

With spoken languages, you can search by spelling or pronunciation. But the "form" of a sign is defined across multiple dimensions: handshape, location, direction of movement, and more. It's difficult for most people to translate a remembered gesture into a formal feature code for a database query. SignTrace targets exactly this reverse-lookup problem, providing natural language access to a Chinese sign language dictionary.

SignTrace's Technical Architecture

According to the paper, SignTrace is not a single model but a complete retrieval pipeline covering everything from dictionary enrichment to candidate reranking. Its core components include:

Five Processing Stages

• LLM-based dictionary enrichment: Large language models enrich dictionary entries with richer descriptions, compensating for overly sparse original entries.
• Action extraction: Key action elements are extracted from the user's natural language description.
• Dictionary-style rewriting: The user's informal description is rewritten into a form closer to standardized dictionary language, narrowing the semantic gap during retrieval.
• Seven-channel retrieval: Candidate entries are retrieved in parallel across seven dimensions, maximizing recall coverage.
• Candidate reranking: Retrieved candidates are reordered to surface the most relevant match at the top.

The system covers 6,699 dictionary entries — a scale sufficient for practical sign language lookup.

---

The seven-channel retrieval design is worth a closer look. A single sign can be described from multiple semantic angles — handshape (e.g., fist, open hand, V-sign), movement trajectory (upward, circular, side-to-side), body location (chest, neck, face), symmetry, and so on. A single retrieval channel can only capture one or two of these dimensions, risking missed results. Seven parallel channels effectively query the dictionary from seven distinct feature perspectives simultaneously, then merge results — so that any valid detail the user happens to mention has a chance of being captured by at least one channel. This dramatically improves recall coverage and is what makes the subsequent reranking stage effective: first ensure the correct answer is in the candidate pool, then rerank it to the top.

Retrieval Performance: Reranking Makes a Major Difference

On a benchmark test set derived from the dictionary (containing 500 action description queries), SignTrace delivers impressive results:

• Hit@1: 94.0% — the correct answer appears as the top result
• Hit@9: 97.4% — the correct answer appears within the top nine results
• Mean Reciprocal Rank (MRR): 0.9540 — approaching 1.0, indicating the correct answer consistently ranks near the top

The contribution of the reranking stage is striking. The data shows reranking boosted Hit@1 from 71.8% to 94.0% — a gain of over twenty percentage points. Without reranking, the first-result accuracy would be dramatically lower. This comparison clearly demonstrates how critical candidate reranking is to the overall pipeline. Ablation analysis also confirmed that enriched dictionary descriptions positively contribute to retrieval quality.

On the performance side, with six concurrent queries, the system's median query processing time is 13.37 seconds. For a multi-stage pipeline involving LLM calls, this latency is acceptable, though there is still room for optimization toward more immediate responses.

---

Mean Reciprocal Rank (MRR) is a standard metric in information retrieval. For each query, the system finds the rank position r of the first correct answer and takes its reciprocal 1/r; MRR is the average across all queries. If the correct answer is always ranked first, MRR = 1.0; if always second, MRR = 0.5. SignTrace's MRR of 0.9540, combined with a Hit@1 of 94.0%, indicates that in the vast majority of queries the correct answer appears at position one or two. MRR is especially important for user experience — users typically only carefully examine the first few results, so having the right answer near the top matters more than simply having it somewhere in a long list.

Practical Value and Limitations

SignTrace has been tried by users who provided positive informal feedback. Its significance lies in bridging the gap between "everyday motion descriptions" and "documented signs with defined meanings," offering a genuinely workable path to identifying unfamiliar signs. For sign language learners, educators, and researchers, the application scenarios are clear.

However, the authors candidly acknowledge the limitations of the current evaluation. The benchmark queries were derived from dictionary-style phrasings and involve prior selection, which may cause the system's generalization ability to degrade when facing entirely spontaneous, user-generated descriptions. In other words, the 94.0% Hit@1 on the benchmark was achieved under relatively "standardized" description conditions; real users describing signs off the cuff tend to use much more informal and ambiguous language, raising the retrieval difficulty considerably.

This also points toward future research: making the system better at understanding the free-form, highly varied natural language descriptions of ordinary users — not just descriptions close to dictionary conventions — will be a critical step toward broader real-world deployment.

---

The prior selection problem mentioned in the paper is a common source of evaluation bias in information retrieval research. When benchmark queries are rewritten from the dictionary's own descriptions, they naturally align closely with the dictionary's language style — essentially competing on home turf. Real users describing an unfamiliar sign tend to use far more casual and subjective language, such as "both hands rubbing together like washing hands" or "right hand shaking next to the face for a moment," which is semantically much further from standardized dictionary phrasing than the benchmark simulates. This gap between the training/evaluation distribution and the real-world usage distribution is the core risk that needs to be validated through genuine user studies.

Summary

SignTrace demonstrates a pragmatic application of large language models in domain-specific retrieval — not as a showcase of technical flash, but as a genuine solution to a long-overlooked practical problem. Through the combination of LLM enrichment, multi-channel retrieval, and reranking, it achieves strong quantitative results on the sign language reverse-lookup task. While its generalization to real-world conditions remains to be validated, it offers a valuable reference paradigm for accessibility tools and language learning applications.