Multimodal Collaborative Agents: A Complete Methodology from Vague Intent to Precise Recommendations

At the AI Engineer conference, Nidhi, Product Lead at Google DeepMind, shared a highly practical topic—Multimodal Collaborative Agents. The core challenge these agents address isn't executing clear instructions, but rather dealing with users' "vague intentions": when users haven't fully figured out what they want or lack precise keywords, how can agents proactively guide, progressively explore, and ultimately help users reach their goals?

Multimodal refers to AI systems that can simultaneously process and understand multiple types of input—text, images, audio, video, etc. Traditional chatbots only handle text, while multimodal agents allow users to send a picture, speak audio, or even share a video, and the system understands the semantics. Google DeepMind's representative work in this field includes the Gemini series models, designed as natively multimodal from the architecture level. "Collaborative" means the agent doesn't just execute commands unidirectionally, but forms a bidirectional partnership with users—in Human-Computer Interaction (HCI) research, this is called "mixed-initiative interaction," where both system and user can proactively initiate dialogue, make suggestions, or request clarification.

This article uses e-commerce shopping as an example (these patterns apply equally to vertical consumer domains like finance and education) to break down a complete methodology from "vague intuition" to "precise recommendations."

Why Current Shopping Agents Can't Handle Vague Intent

Most shopping agents on the market today are essentially "a wrapper to the search bar." They have an implicit assumption: users already have clearly defined intent, master the right keywords, and know what they're looking for.

But reality differs. Nidhi points out a massive articulation gap between users and agents. When shopping, users rarely arrive with fully formed needs—they come with a "vague feeling" or "vibe."

This concept has deep academic roots. In information retrieval, Nicholas Belkin proposed the "Anomalous State of Knowledge (ASK)" hypothesis in the 1980s: when users initiate search behavior, they're often in an "anomalous state" of knowledge—they know they lack certain information but cannot precisely describe what's missing. In e-commerce, this gap is especially prominent—research shows over 60% of online shopping journeys begin with "browsing" rather than "searching" behavior. Users don't have clear purchase goals but gradually form preferences through exploration. This explains why simple keyword search boxes can't meet these needs.

This requires agents to do significant "hand-holding" work: first understand or even proactively excavate user preferences, then show users various possibilities, and finally progressively converge toward the most suitable recommendations within user constraints.

The entire process forms a flywheel loop from "extremely vague intent" toward "achieving user goals," encompassing three core stages: Discovery, Research, and Response.

Discovery Phase: Identifying Truly Important User Signals

The discovery phase's key is having the agent identify "what information truly matters." The agent extracts context from multiple signal sources: past conversations, reference images or links provided by users, personal context, etc., and builds a "working state" accordingly.

Taking "user wants to redecorate living room within a budget" as an example, the agent's working state includes conversation history, user context, and extracts hard constraints (like budget) from the query.

The truly interesting part is handling soft constraints. When users can't verbally describe needs and only provide an "inspiration reference image," the agent needs to more proactively extract salient signals from these multimodal inputs, build a mental model of user preferences, and assign a confidence score to each inference.

Applying confidence scores is essentially implementing Bayesian Inference thinking. Each inference the agent makes about user preferences comes with a confidence value between 0 and 1, representing the agent's confidence in that inference's correctness. As conversation progresses and users provide more signals, these confidence values dynamically update based on new evidence—this is exactly the Bayesian theorem process where "prior probability" updates to "posterior probability" with "observed data." For example, if the agent infers from a reference image that the user prefers "minimalist style" with 0.7 confidence, when the user later clicks on a Scandinavian furniture image, this confidence might rise to 0.85. This probabilistic modeling, compared to traditional binary judgments (yes/no), handles ambiguity and multiplicity in user preferences more elegantly.

Additionally, the agent needs to refresh certain variables in real-time—like inventory. If recommendations use outdated information, the recommendations lose meaning.

How to Discover Intent Gaps and Increase Information Gain

The second step in discovery is finding the "intent gap." Before providing the best answer, many unknown variables exist, but the agent doesn't need to ask about every unknown.

The key is formulating a collaborative strategy: compare all possible "next steps" and prioritize asking the question that brings maximum information gain.

Information Gain is a core concept in information theory, originating from Claude Shannon's entropy theory. In agent decision frameworks, information gain measures: after asking a question, how much can the agent reduce uncertainty about user intent? Mathematically, this equals the difference in conditional entropy before and after the question is answered. This thinking highly relates to the "Active Learning" paradigm in machine learning—in active learning, models actively select the most valuable samples to request labeling, achieving maximum model improvement with minimal labeling cost. Nidhi applies this thinking to dialogue strategy: among many possible follow-up questions, the agent chooses the one that most eliminates uncertainty, reaching the user's true needs with the fewest dialogue turns.

For instance, in the renovation case, "room width" might be the optimal next question—because if recommended furniture can't fit in the room, everything is futile. This variable will substantially change the dialogue direction.

At the evaluation level, Google DeepMind uses auto-raters, focusing on: whether the agent identifies all necessary "blockers," whether it "over-asks," and "question utility"—whether each question truly helps excavate correct preferences.

Research Phase: Using Visual Display Instead of Text Questions

The research phase's core insight: not all preferences are suitable for text questions.

When the agent explores subjective constraints like users' "style preferences," it must first build a temporary bridge between the constraint itself and the product catalog and knowledge base ontology, enabling subsequent retrieval to map constraints back to the knowledge base.

Ontology is a core concept in knowledge engineering, referring to formal descriptions of concepts, attributes, and their relationships within a domain. In e-commerce, product ontology defines category hierarchies (like "furniture → sofa → sectional sofa"), attribute dimensions (material, color, size, style), and associations between attribute values (like "Scandinavian style" typically associates with "light wood" and "minimalist lines"). Building a "temporary bridge" from users' vague descriptions to ontology's structured attributes is essentially semantic alignment—mapping user statements like "looks cozy" to specific attributes in the ontology like "warm tones," "soft materials," and "rounded shapes." Before large language models emerged, this mapping mainly relied on manual rules and thesauri; today LLM semantic understanding makes this mapping more flexible and robust.

For subjective, ambiguous constraints like style, text questions often aren't the optimal solution. The agent chooses to guide users with a visual preference board.

Visual preference board design traces inspiration to the "Mood Board" methodology widely used in design—designers align aesthetic direction with clients by collaging images, colors, and material samples. In cognitive psychology, this leverages humans' "Recognition over Recall" cognitive characteristic: humans find it far easier to recognize what they like from options than to describe preferences from scratch using language. By showing visual options rather than asking for text descriptions, agents essentially transfer cognitive burden from users' "generative thinking" to "judgmental thinking," dramatically lowering users' expression costs. Simultaneously, visual boards help users "discover what they didn't know they wanted"—in recommender systems research, this is called "Serendipity."

The specific approach: based on existing constraints, conversation history, and temporary mapping, generate several style options most similar to the user's reference image, establishing "common language" between user and agent.

Further, the agent observes users' micro signals—like mouse hover, click direction—and uses these to update the confidence model of user style preferences in real-time.

Micro signals like mouse hover duration, scroll speed, and click order belong to "Implicit Feedback" in recommender systems. Unlike ratings or reviews actively provided by users (explicit feedback), implicit feedback is behavioral data users inadvertently produce, yet often more authentically reflects user preferences. Early Netflix research found users' actual viewing behavior predicts future preferences better than ratings they give. In agent scenarios, if a user hovers on a sofa image for 3 seconds while spending only 0.5 seconds on other images, this micro signal's information content might exceed users actively saying "I like this"—because it hasn't passed through users' rational filtering, coming closer to instinctive reaction. Of course, micro signals carry more noise, requiring weighted integration with other signals—this is where real-time confidence model updates provide value.

Three-Dimensional Evaluation System for Research Phase

Auto-raters for this phase focus on three dimensions:

• Hidden preference discovery efficiency: Use user simulators to inject constraints and observe the agent's efficiency in excavating these constraints. User simulators are an important innovation in AI agent evaluation—traditional agent evaluation relies on real user testing, which is costly, time-consuming, and hard to standardize. User simulators use large language models to play "virtual users" with specific preferences and behavior patterns, enabling large-scale, automated agent testing. In Google DeepMind's approach, simulators are pre-injected with specific hidden constraints (like "prefers blue," "budget under 5000"), then observe how many dialogue turns the agent needs to excavate these constraints. This method borrows from "Agenda-based User Simulation" frameworks in dialogue systems research, but LLM inclusion makes simulator responses more naturally diverse.
• Turn efficiency: How many dialogue turns reveal all hidden preferences, avoiding circular loops;
• Format selection accuracy: Whether the agent used the right question format—use text for easily stated questions, visual anchors for ambiguous questions with articulation gaps.

Response Phase: Adaptively Shaping Answer Presentation

When the agent already grasps style preferences, budget constraints, and relevant dimensions, the final step is deciding how to present the answer. This is where many recommendation systems fail—they just dump a wall of text.

Nidhi emphasizes, response format itself is part of intelligence. Agents should adaptively select presentation based on query type:

• Querying product policies or reviews → use summaries or bullet lists;
• Comparing two products → use tradeoff tables / comparison tables;
• Seeking style inspiration → use visual reference images and product inspiration photos.

Auto-raters evaluating the response phase focus on three points: format accuracy (information shouldn't be buried, users should see it at a glance), data fidelity (models can't hallucinate, information must be accurate and complete), and user actionability (response format should give users confidence to proceed to the next step—completing purchase).

Emphasizing data fidelity directly addresses one of large language models' most criticized problems—hallucination. Hallucination means model-generated content appears fluent and reasonable but actually contains fabricated facts or data. In e-commerce recommendation scenarios, hallucination costs are especially severe: if an agent incorrectly claims a sofa "supports free returns" or "has a 4.8-star rating" when reality differs, it not only damages user trust but may trigger legal risks. Current mainstream industry solutions for hallucination include Retrieval-Augmented Generation (RAG)—having models generate answers based on retrieved real product data; fact-checking layers—automatically comparing generated content against source databases before output; and citation tracing—annotating information sources in answers. Google DeepMind's auto-raters specifically including a data fidelity dimension shows they treat this as a hard threshold for product usability.

Four Core Design Principles for Collaborative Agents

Nidhi finally distills the entire methodology into four actionable design principles:

1. Be ready to receive "vibes": Users come with vague intentions; systems must handle messy queries;
2. Show and ask: Visuals and comparisons reveal preferences faster than text, letting agents and users speak "the same language";
3. Shape the answer: Presentation format should help users quickly find key information; the model's response structure is part of intelligence;
4. Grade the loop: Establish auto-raters at every step, and this scoring system should evolve with the system—start simple, grow progressively.

Merchant Ontology Structure and Human-Machine Collaboration Engineering Considerations

During Q&A, Nidhi revealed several noteworthy engineering and product judgments:

On merchant ontology structure, Google DeepMind heavily leverages merchants' domain expertise to build "constraint-metadata" mapping bridges, combined with the recently launched UCP protocol, enabling merchants to speak "the same language" as agents.

UCP (Unified Commerce Protocol) is a standardization protocol recently launched by Google, aiming to establish unified data interaction specifications between AI agents and e-commerce systems. Similar to APIs' role between software systems, UCP defines standardized formats and interaction flows for data like product information, inventory status, pricing strategies, and return policies. This echoes industry movements like Shopify's MCP (Merchant Commerce Protocol), reflecting an emerging industry consensus: when AI agents become new shopping entry points, merchants need a standardized way to "describe themselves" to agents. Without such standardized protocols, each agent needs custom integration for each merchant, clearly unscalable.

However, response format decision-making currently belongs entirely to agents, with no merchant control—the goal is establishing a unified horizontal experience layer across all merchants.

Regarding human-machine division of labor, user research revealed an interesting phenomenon: at the upper funnel (discovery, inspiration, exploration stages), users prefer personal participation and deep interaction; whereas at the lower funnel (price comparison, evaluation, negotiation), users are more willing to delegate to agents.

The shopping funnel is a classic marketing model, typically including "awareness → interest → consideration → intent → evaluation → purchase" stages. Nidhi's finding challenges many AI entrepreneurs' "full automation" visions: users actually prefer personal involvement in the upper funnel. This aligns with psychology's "Need for Autonomy" theory—humans naturally want control when making decisions involving personal taste and values. This finding also echoes MIT researcher Erik Brynjolfsson's "Augmentation vs. Automation" framework: the most successful AI applications don't replace human decisions but augment human decision-making capabilities. In actual product design, this means agents should play "inspiration catalyst" and "information organizer" roles during exploration, not "decision proxy"; only in downstream tasks where users explicitly express delegation intent should they switch to full automation.

This finding has reference value for all teams building consumer-grade agents—don't try to do everything for users, especially during exploration.

Key Takeaways

• Multimodal collaborative agents address the articulation gap where users arrive with vague "vibes" rather than clear requirements
• The three-stage flywheel—Discovery, Research, Response—progressively narrows from ambiguity to precision
• Discovery phase: Build confidence-scored user preference models, prioritize questions by information gain
• Research phase: Use visual preference boards instead of text questions for subjective constraints like style
• Response phase: Adaptively shape answer format based on query type—summaries, comparison tables, or visual galleries
• Four design principles: accept vibes, show and ask, shape answers, grade the loop
• User research reveals upper-funnel preference for human control, lower-funnel acceptance of agent delegation