ViSkill
A vision-native skill learning framework that encodes successful interactions as composite visual skill cards for direct use by VLM agents in reasoning and reward shaping, achieving mutual reinforcement between the skill library and the policy through closed-loop feedback.
Timeline (last 90 days)
ViSkill is a vision-native skill learning framework that lets VLM agents encode successful interactions as composite visual skill cards, call them directly during inference and reward shaping, and through closed-loop feedback let the skill library and policy reinforce each other.
ViSkill achieves an overall success rate of 0.89 on the three environments Sokoban, FrozenLake, and PrimitiveSkill, rising to 0.91 when cold start is enabled.
ViSkill stores skill cards in visual form and feeds them to the VLM as image context during inference, letting the model's own visual encoder process geometric structure, rather than first translating spatial relations into text through an external pipeline.
The upper bound of the ViSkill framework is limited to some extent by the spatial perception ability of the chosen VLM.
The additional reward ViSkill uses for reward shaping comes from retrieved visual skill cards, rather than manually designed heuristic functions.
ViSkill reports that its performance exceeds all evaluated proprietary and open-source baselines, while converging faster than standard PPO.
ViSkill's code has been open-sourced on GitHub, with the repository being ZJU-REAL/ViSkill.
ViSkill provides an optional cold-start mechanism to accelerate learning in the early stage, allowing the agent to get started relatively quickly even when the skill library is not yet rich.
All Facts (8)
ViSkill provides an optional cold-start mechanism to accelerate learning in the early stage, allowing the agent to get started relatively quickly even when the skill library is not yet rich.
50%UnverifiedViSkill achieves an overall success rate of 0.89 on the three environments Sokoban, FrozenLake, and PrimitiveSkill, rising to 0.91 when cold start is enabled.
50%UnverifiedViSkill stores skill cards in visual form and feeds them to the VLM as image context during inference, letting the model's own visual encoder process geometric structure, rather than first translating spatial relations into text through an external pipeline.
50%UnverifiedThe upper bound of the ViSkill framework is limited to some extent by the spatial perception ability of the chosen VLM.
50%UnverifiedThe additional reward ViSkill uses for reward shaping comes from retrieved visual skill cards, rather than manually designed heuristic functions.
50%UnverifiedViSkill reports that its performance exceeds all evaluated proprietary and open-source baselines, while converging faster than standard PPO.
50%UnverifiedViSkill is a vision-native skill learning framework that lets VLM agents encode successful interactions as composite visual skill cards, call them directly during inference and reward shaping, and through closed-loop feedback let the skill library and policy reinforce each other.
50%UnverifiedViSkill's code has been open-sourced on GitHub, with the repository being ZJU-REAL/ViSkill.
50%