74% of Technical Users Want AI Agents to Confirm Before Acting: Where Is the Boundary Between Autonomy and Control?

AI Agent Behavior Preferences Revealed by a Small Survey

Anastasia, an intern on the open-source AI agent runtime project HugiMuni and a master's student at the Technical University of Munich (TUM), recently conducted a small survey within her technical network. The sample size was only 27-28 people (depending on the question), and it was not a random sample, so the results should be treated as early signals rather than representative research. Still, a few data points are worth noting.

When asked "If the AI thinks the user's request is a bad idea, what should it do?", 74% (20/27) of respondents chose ask for confirmation first; 33% (9/27) chose "proceed but flag the risk"; 30% (8/27) chose "push back and suggest a better approach"; and only 7% (2/27) chose "just execute." Multiple selections were allowed for this question.

These figures sketch a clear tendency: technical users are not opposed to AI's autonomous judgment, but they want a confirmation step retained whenever the AI deviates from user expectations.

What Users Most Want to Shape Is Judgment, Not Communication Style

The survey also asked which aspects of AI behavior respondents most wanted to shape. 56% prioritized judgment—how the AI makes decisions, when it should ask, when it should act, and how bold it should be. 44% chose candor, i.e., how honestly the AI challenges the user. 33% focused on code quality standards. Only 4% chose communication style.

Another noteworthy result: 56% of respondents preferred writing their own instructions rather than choosing from preset AI personas.

This set of preferences reflects a deeper need: technical users treat AI agents as tools that require fine-tuning, not as consumer products where you pick a "personality." What they care about is decision logic and quality baselines, not whether the AI sounds likable.

Autonomous Execution Does Not Mean Unrestricted Authorization

The HugiMuni team raised a core tension in their post: we want agents to autonomously investigate problems, run tests, and keep working without constant supervision, yet we don't want to give them unconstrained permissions.

An agent fixing a small bug with good test coverage is one thing; an agent deciding to refactor the production database is another. The truly difficult engineering question is: where should this boundary be drawn?

Should it be defined through persistent user instructions? Or explicit permissions? Tool-level policies? Automated verification? Or some combination? And how should these boundaries persist across sessions, across projects, and across different AI models?

There are currently no standard answers to these questions. Different agent products (such as Claude Code, Codex, and Cursor) make different trade-offs in their permission models, and users' actual tolerance varies with the risk level of the task.

Agent Runtime refers to the underlying infrastructure that supports an AI agent's continuous operation, tool invocation, state maintenance, and execution of multi-step tasks. Unlike a single-conversation chatbot, an agent runtime must manage session lifecycle, tool permissions, resource access, and error recovery. In the case of open-source projects like HugiMuni, the runtime typically includes a sandbox environment, filesystem access control, a whitelist for command execution, and cross-session state persistence mechanisms. It is precisely these components that determine the actual boundary between "what the agent can do" and "what the agent is allowed to do," and they are the technical vehicle through which permission models are implemented. Only by understanding the division of responsibilities in the runtime can one understand why permission boundaries are hard to define simply—they simultaneously involve trade-offs at the infrastructure layer, the policy layer, and the user experience layer.

Tool-level Policy is a granularity choice for agent permission control: rather than authorizing at the session or user level as a whole, rules are set individually for each callable tool (such as reading/writing files, executing shell commands, or making network requests). For example, an agent could be allowed to freely read the codebase, but require explicit confirmation to write to production configuration files, and be completely prohibited from executing database migration commands. The advantage of this approach is that boundaries are clear and auditable, but the downside is that policies become complex as the number of tools grows, and it is difficult to cover risks arising from tool combinations (such as reading a secret and then exfiltrating it). Therefore, tool-level policies usually need to be combined with automated verification (such as command whitelists and diff review) to form an actionable protection system.

Moving from Hypothetical Preferences to Real Experience

The HugiMuni team explicitly stated that this survey aggregates anonymous results and serves only as an early signal, not as evidence of broader market preferences. Their next research direction is to move from "hypothetical preferences" to "real usage experience," focusing especially on developers who use agents such as Claude Code, Codex, and Cursor daily: Which operations are you comfortable letting the agent do directly without asking? Which operations should it never perform without confirmation? Are there instances where an agent truly overstepped?

For teams building agent runtimes, these real-world cases are more instructive than survey data. The balance point between autonomy and control will ultimately need to be repeatedly calibrated in specific workflows and risk scenarios.

One-line Summary

A small survey shows technical users want AI agents to confirm before deviating from expectations, valuing judgment over communication style.

Paragraph Summary

Anastasia, an intern on the HugiMuni team, conducted a non-random small-sample survey of 27-28 technical users, with results serving as early signals rather than representative conclusions. The survey found that 74% of respondents want AI to ask for confirmation first when it deems a request inappropriate, while only 7% accept direct execution; 56% most want to shape AI's judgment, and only 4% focus on communication style; 56% prefer writing their own instructions rather than choosing preset personas. The team thus raises a core engineering tension: how to define the boundary between autonomous agent execution and unrestricted authorization, and make it persist across sessions. The next phase of research will turn to real usage experience, focusing on actual overstepping cases among developers using agents such as Claude Code, Codex, and Cursor.