Trek-Memory
A smart home instruction evaluation dataset extended from HomeBench. It introduces multi-turn dialogue history and requires resolving pronoun references and ellipsis, thereby assessing dialogue memory capability.
Timeline (last 90 days)
The Trail team extended HomeBench into three datasets: Trek-Base (testing language diversity), Trek-Robust (evaluating robustness to environment changes), and Trek-Memory (evaluating multi-turn dialogue memory capability).
With the memory module, Trail can solve 75.6% of requests that depend on previous dialogue turns, whereas the fine-tuned baseline achieves only 42.5% when given the dialogue history.
All Facts (2)
With the memory module, Trail can solve 75.6% of requests that depend on previous dialogue turns, whereas the fine-tuned baseline achieves only 42.5% when given the dialogue history.
50%UnverifiedThe Trail team extended HomeBench into three datasets: Trek-Base (testing language diversity), Trek-Robust (evaluating robustness to environment changes), and Trek-Memory (evaluating multi-turn dialogue memory capability).
50%