Humans Present but Powerless: Why AI Agent "Human-in-the-Loop" Is a Rubber Stamp

AI agents' safety bottom line has long rested on "Human-in-the-Loop" — having humans review and approve every critical decision the AI makes. But three AI ethics researchers at Hugging Face argue in a paper published on ArXiv that unless designers and users change their current practices, this mechanism is doomed to fail, and may even truly push humans "out of the loop."

Why "Human-in-the-Loop" Has Become a Formality

Avijit Ghosh, Principal Technical AI Policy Researcher at Hugging Face and one of the paper's authors, offers a sharp analogy: in current practice, "the human just becomes a meat tool that grants permissions, with no cognitive capacity to genuinely participate."

The root of the problem lies in the agents' design objectives themselves. The authors point out that today's agents are tuned to satisfy benchmarks like speed, accuracy, and throughput, while the needs of human supervisors are treated as "a separate consideration independent of system quality." The result is that agents generate far more information than humans can possibly comprehend.

A real-world example is the Hugging Face attack incident: roughly 1,200 bots produced 1.2 million messages in a hastily built communication system — no human supervisor could possibly digest them all. According to the paper's three authors (besides Ghosh, Hugging Face Chief Ethics Scientist Margaret Mitchell and Samir Passi of the Institute for Data & Society), although the paper was written before the incident was disclosed, the conclusion was precisely "what would happen if current trends continued, based on logical extrapolation — and we saw it all accelerated and validated through the Hugging Face attack."

Human-in-the-Loop (HITL) is a concept originating from cybernetics, human-computer interaction, and automated system design, referring to mechanisms that retain human participation at key decision points in automated processes. It was first applied in aviation autopilot, industrial process control, and medical diagnostic systems, with the core assumption being that human judgment can correct machine errors, especially in uncertain or high-risk scenarios. HITL typically operates at several levels: human-in-the-loop (humans approve each case), human-on-the-loop (humans monitor and can intervene), and human-out-of-the-loop (fully automated). This article criticizes precisely how the first level degrades in practice — the approval button exists, but cognitive engagement is absent, forming "nominal HITL." Understanding this spectrum helps distinguish between "having an approval action" and "having substantive oversight."

How Cognitive Biases Dismantle Oversight

The paper deeply analyzes the structural weaknesses of human cognition when facing AI suggestions — flaws that cannot easily be overcome through training:

• Automation bias: Users accept system recommendations even when they are wrong
• Anchoring bias: People tend to simply agree with AI decisions rather than thinking through alternatives
• The disutility of effortful reasoning: Deep thinking is psychologically far less comfortable than quickly approving an AI proposal, especially under information overload
• AI's sycophantic tendency: AI tends to tell users "you did well," which precisely undermines the skepticism and self-monitoring that oversight requires

These biases compound into a kind of "cognitive surrender": the more users rely on agents, the more they lose the cognitive capacity needed to control them — a self-reinforcing vicious cycle.

The Logical Flaw of "Using AI to Supervise AI"

Ghosh specifically criticizes a popular practice: using another large language model to monitor an agent's behavioral logs. His rhetorical question cuts to the heart of it: "How do we know the two LLMs won't collude?"

This concern is not paranoid. If the supervisor and the supervised share similar training objectives, architectural biases, and blind spots, then so-called "independent oversight" is merely an illusion.

You may not have noticed, but researchers in autonomous driving and robotics have been deeply engaged with human-machine interaction issues for decades. Mary L. Cummings, Director of the Center for Autonomous and Robotic Systems at George Mason University, commented in an email to IEEE Spectrum that AI developers are "late to the party" on this issue — they have lagged far behind adjacent fields in paying attention to "cognitive engineering." She also noted that the paper "uses a lot of academic vocabulary to express one point: AI companies should care about human factors."

LLM collusion in the AI safety context does not refer to two models consciously conspiring, but rather to their generation of highly correlated blind spots and biases due to shared training data distributions, objective functions, and architectural priors. When the supervisor and the supervised share the same origin, the supervisor is likely equally insensitive to the same types of errors, or may even rationalize them as correct. This "correlated failure" has been discussed in financial risk control and multi-agent reinforcement learning: multiple seemingly independent models may fail simultaneously due to homogenization. Ghosh's challenge essentially demands that supervisors possess statistical independence — different training objectives, data sources, or architectures are needed to provide genuinely complementary judgment; otherwise, "independent oversight" merely repeats the same error.

Getting Humans Truly Involved: Introducing "Friction"

The paper's core recommendation is to deliberately introduce "friction" into human-AI interaction to prevent users from falling into boredom, passivity, or unconscious clicking. Specific proposals include:

1. Commit before revealing: The agent asks users to record their own judgment before showing its plan
2. Counterintuitive follow-up questions: When a user approves, the agent asks back, "What evidence would change your mind?"
3. Behavior-adaptive responses: If the system detects that the time a human spends on each approval is decreasing, the agent proactively changes its behavioral pattern
4. Institutional fatigue prevention: Organizations should arrange for employees to periodically complete tasks independently of agents, or require mandatory breaks

These suggestions all introduce delays — precisely what agents are supposed to "reduce." But Ghosh argues that when humans cannot monitor and control AI, "productivity gains are a myth." The time saved by delegating to agents must be weighed against the time spent fixing agents' errors.

Mitchell's statement on social media captures the essence of the problem: "Safety and capability should not be separate. It only slows things down when safety is bolted on rather than being part of the core technology."

For teams currently bringing agents into production environments, this implies an uncomfortable conclusion: genuine "human-in-the-loop" is not about adding an approval button, but about redesigning the cognitive division of labor between humans and AI.

Cognitive Engineering is an interdisciplinary field proposed by Donald Norman and others in the 1980s, studying how to design technological systems to match human limitations in perception, memory, attention, and decision-making. It emphasizes "human-centered" design: not requiring people to adapt to machines, but making machines adapt to human cognitive capacity. Mary L. Cummings points out that AI developers are "latecomers" to this field, meaning many AI teams reduce human oversight to an approval interface without systematically considering classic human factors issues such as attention fatigue, trust calibration, and situational awareness. Safety-critical industries like aviation, nuclear power, and healthcare have long established mature human factors engineering methods — such as mandatory checklists, dual review, and simulation training — and AI agent design can draw on this accumulated knowledge rather than trial-and-error from scratch.

One-line Summary

Hugging Face researchers warn: current "human-in-the-loop" design is pushing humans out of the oversight role and needs deliberate friction.

Paragraph Summary

Three AI ethics researchers at Hugging Face point out that the current "human-in-the-loop" mechanism for AI agents has become a mere formality in practice. Agents prioritize speed and efficiency, treating human supervisors as a secondary factor, resulting in information volumes far exceeding human processing capacity. Automation bias, anchoring bias, and AI sycophancy together trigger "cognitive surrender" — the more users rely on AI, the more they lose oversight capacity. Using another LLM to supervise AI also carries collusion risks. The paper recommends restoring substantive human participation through "friction" designs such as commit-before-reveal, asking for evidence, behavior-adaptive responses, and institutional fatigue prevention. The authors emphasize that safety and capability should not be separated, and genuine oversight is not about adding an approval button, but redesigning the cognitive division of labor.