PSGAgent
An agent framework for panoptic scene graph generation (PSG) that uses a think-act-observe loop to let a multimodal large language model dynamically invoke region zoom-in and visual highlighting tools, overcoming the perceptual limits of a static observer.
Timeline (last 90 days)
The researchers proposed PSGAgent, an agent framework that enhances PSG visual perception through a "think-act-observe" loop.
The PSGAgent framework includes a Region Zoom-In Tool for zooming in on small objects or boundary relations to obtain higher-resolution local views.
The PSGAgent framework includes a Visual Highlight Tool that filters irrelevant contextual noise through relation-oriented segmentation, focusing the model on foreground objects and relations relevant to the current reasoning.
PSGAgent adopts a two-stage training pipeline: the first stage builds an error-aware multimodal chain-of-thought (M-CoT) dataset for cold-start learning; the second stage uses a hierarchical reward mechanism combined with group sequence policy optimization (GSPO) to learn the tool-use policy.
PSGAgent achieves SOTA performance on the PSG benchmark, demonstrating strong capabilities in both overall and fine-grained scene understanding.
The significance of PSGAgent extends beyond the PSG task itself: it offers a paradigm that multimodal large models can draw on in scenarios requiring fine-grained visual reasoning—teaching models to actively perceive rather than passively receive.
All Facts (6)
The PSGAgent framework includes a Region Zoom-In Tool for zooming in on small objects or boundary relations to obtain higher-resolution local views.
50%UnverifiedThe PSGAgent framework includes a Visual Highlight Tool that filters irrelevant contextual noise through relation-oriented segmentation, focusing the model on foreground objects and relations relevant to the current reasoning.
50%UnverifiedPSGAgent adopts a two-stage training pipeline: the first stage builds an error-aware multimodal chain-of-thought (M-CoT) dataset for cold-start learning; the second stage uses a hierarchical reward mechanism combined with group sequence policy optimization (GSPO) to learn the tool-use policy.
50%UnverifiedPSGAgent achieves SOTA performance on the PSG benchmark, demonstrating strong capabilities in both overall and fine-grained scene understanding.
50%UnverifiedThe significance of PSGAgent extends beyond the PSG task itself: it offers a paradigm that multimodal large models can draw on in scenarios requiring fine-grained visual reasoning—teaching models to actively perceive rather than passively receive.
50%UnverifiedThe researchers proposed PSGAgent, an agent framework that enhances PSG visual perception through a "think-act-observe" loop.
50%