EgoOmniWorld
A unified methodological framework for egocentric video understanding whose core principle is the Ego-Action-World interaction loop. It decomposes interactions into four complementary structural levels: Node, Edge, Chain, and Loop. The framework adopts an interaction-first data construction approach to generate multimodal question-answer pairs.
Timeline (last 90 days)
The paper proposes EgoOmniWorld, an egocentric video understanding framework unified by the Ego-Action-World interaction loop as its principle.
EgoOmniWorld decomposes interaction into four complementary structural levels, Node, Edge, Chain and Loop, with progressively increasing complexity.
EgoOmniWorld adopts an interaction-first data construction approach: structured interaction information is extracted from videos, anchored to audiovisual evidence, and then question-answer pairs are automatically generated.
EgoOmniWorld contains 24,166 QA groups, including 20,284 training samples and 3,882 test samples.
The paper evaluates on multiple open-source and closed-source multimodal large language models and finds that model performance across different interaction structures varies significantly and is uneven.
Models trained on EgoOmniWorld not only improve in-domain interaction understanding, but also generalize to existing egocentric benchmarks.
EgoOmniWorld's data, code and models will be released open source.
EgoOmniWorld's evaluation targets multimodal large language models capable of processing video, audio and text simultaneously.
The training signal of interaction structure has general value and is not only effective for a specific evaluation set.
All Facts (9)
The paper proposes EgoOmniWorld, an egocentric video understanding framework unified by the Ego-Action-World interaction loop as its principle.
50%UnverifiedEgoOmniWorld decomposes interaction into four complementary structural levels, Node, Edge, Chain and Loop, with progressively increasing complexity.
50%UnverifiedEgoOmniWorld adopts an interaction-first data construction approach: structured interaction information is extracted from videos, anchored to audiovisual evidence, and then question-answer pairs are automatically generated.
50%UnverifiedEgoOmniWorld contains 24,166 QA groups, including 20,284 training samples and 3,882 test samples.
50%UnverifiedThe paper evaluates on multiple open-source and closed-source multimodal large language models and finds that model performance across different interaction structures varies significantly and is uneven.
50%UnverifiedModels trained on EgoOmniWorld not only improve in-domain interaction understanding, but also generalize to existing egocentric benchmarks.
50%UnverifiedEgoOmniWorld's evaluation targets multimodal large language models capable of processing video, audio and text simultaneously.
50%UnverifiedThe training signal of interaction structure has general value and is not only effective for a specific evaluation set.
50%UnverifiedEgoOmniWorld's data, code and models will be released open source.
50%