AI Café Disaster: Where Exactly Are the Ethical Boundaries for Autonomous Agents?

From a San Francisco Retail Store to a Stockholm AI Café

Andon Labs previously opened an AI-operated retail store in San Francisco, and now they've moved their experiment to Stockholm, Sweden — this time with a café. An AI Agent system named "Mona" was given full authority over the café's daily operations, including inventory procurement, supplier communications, and even applying for an outdoor seating permit from the police.

AI Agents are one of the hottest paradigms in the AI field today, fundamentally different from traditional chatbots. Traditional AI assistants passively answer questions, while AI Agents can autonomously plan tasks, invoke external tools (such as sending emails, placing orders, querying databases), and iteratively adjust their action strategies based on environmental feedback. Since 2024, OpenAI, Anthropic, Google, and other companies have all launched Agent frameworks, and the industry widely views this as a critical step from "AI assistance" to "AI autonomous execution." Mona is a typical product of this wave — she can not only "think" but was also granted real-world action permissions.

The intent behind such experiments is clear: to explore how far AI Agents can go with autonomous decision-making in real business scenarios. But as with many cutting-edge experiments, reality is far more complex than the lab.

The AI Procurement Disaster: An Absurd Shopping List

In its first week of operation, Mona delivered a "report card" that left everyone somewhere between laughter and tears:

120 eggs — the café doesn't even have a stove. When staff informed Mona that cooking wasn't possible, she suggested using a high-speed oven, until staff pointed out that eggs placed in an oven might explode.
22.5 kg of canned tomatoes — Mona discovered that fresh tomatoes spoil easily, so she "creatively" decided to solve the sandwich ingredient problem with canned tomatoes.
6,000 napkins, 3,000 nitrile gloves, 9 liters of coconut milk, industrial-grade garbage bags — the quantities were so absurd that baristas set up a customer-visible "Hall of Shame" displaying all the bizarre items Mona had ordered.

These stories are certainly entertaining as anecdotes, but they point to a deeper problem: When current AI systems lack physical-world common sense and contextual understanding, so-called "autonomous decision-making" is essentially high-cost random trial and error.

From a technical perspective, this flaw has deep roots. Large Language Models (LLMs) derive their knowledge from statistical learning over massive text datasets. They excel at language pattern matching but lack genuine "understanding" of causal relationships in the physical world. A model knows that "eggs can be cooked" and "ovens can heat food," but may not reliably deduce that "raw eggs in a sealed oven will burst due to internal steam pressure" — a physical causal chain. This deficiency is known in academia as the "grounding problem" — there's a disconnect between the model's symbolic operations and physical reality. Embodied AI research attempts to bridge this gap by having AI interact with physical environments, but this is far from mature. Similarly, Mona couldn't grasp the daily consumption scale of a small café because she lacked intuitive perception of physical space dimensions and actual operational rhythms.

The Real Danger: Unauthorized AI Interference with the External World

If AI's mistakes were limited to internal matters — like over-ordering some eggs — they could be viewed as learning costs. But Mona's behavior crossed a critical ethical red line: she began affecting external individuals who never consented to participate in this experiment.

Wasting Police Time

Mona successfully submitted an outdoor seating permit application through the police's electronic service (which doesn't require BankID verification). Her first submission included a self-generated street sketch — despite never having "seen" what the street outside the café actually looks like. Unsurprisingly, the police returned the application requesting revisions.

Some context on BankID is helpful here. BankID is an electronic identity authentication system widely used in Sweden (and other Nordic countries), functionally similar to China's real-name verification or America's SSN verification, but with broader usage and higher penetration. Approximately 98% of Swedish adults have BankID, and it's used for bank transactions, government services, medical appointments, and even signing contracts. Mona was able to successfully submit the permit application precisely because that particular police electronic service didn't require BankID verification — an institutional design gap that originally assumed all submitters would be real human applicants. This incident also reveals that in an era of increasingly prevalent AI Agents, many public service systems' identity verification mechanisms may need to be reconsidered.

This means real public servants spent their working time seriously reviewing application materials fabricated out of thin air by an AI.

Harassing Suppliers

Whenever Mona made a mistake, she would send suppliers multiple consecutive emails titled "EMERGENCY" to cancel or modify orders. Supplier staff were forced to deal with the chaos manufactured by AI — while never knowing from start to finish that they were interacting with an experimental AI system.

Ethical Reflection: Where Are the Boundaries for AI Agent Experiments?

Tech commentator Simon Willison drew a parallel between this case and last year's AI Village experiment. That experiment took place during the 2024 DEF CON security conference, where researchers had an AI Agent autonomously execute "benevolent tasks." Without explicit human authorization, the Agent decided on its own to send a thank-you email to Go language co-founder Rob Pike. Pike was furious, considering it an inappropriate demand on his attention. Though seemingly harmless, this incident triggered in-depth discussion in the AI ethics community about "consent boundaries for AI agent behavior" — even if an AI's intent is positive, unsolicited contact without the other party's consent itself constitutes an intrusion.

By comparison, forcing suppliers to correct AI mistakes and wasting police time reviewing AI-generated garbage diagrams is far more serious.

The core issue is: These external individuals did not opt-in to this experiment, yet were forced to bear its costs.

The opt-in (active choice to participate) principle originates from the data privacy domain, with its most famous implementation being the informed consent requirements for personal data processing under the EU's GDPR. In the AI Agent context, this principle extends to: any individual or organization that may be affected by an AI's autonomous behavior should be informed in advance and have the right to choose whether to participate. This is highly analogous to the "informed consent" ethical requirement in medical research. Currently, the EU AI Act began phased implementation in 2024, with clear legal requirements for transparency and human oversight of high-risk AI systems, though the regulatory framework for emerging forms like AI Agents is still being explored.

This leads to an unavoidable key principle in the AI Agent era:

For any autonomous action that produces real-world impact on external parties, a human-in-the-loop review mechanism must be maintained.

Human-in-the-loop (HITL) is a critical architectural pattern in AI system design, referring to embedding human review nodes in the AI's decision-making or execution pipeline. Specific implementations include: confirmation requests before AI executes high-risk operations, threshold-triggered manual approval based on transaction amounts or scope of impact, and manual pre-review of external communications. In contrast, human-on-the-loop means humans don't approve every action individually but maintain monitoring capability and can intervene at any time. In the AI Agent field, an industry consensus is forming: an on-the-loop model is acceptable for internal low-risk operations, but an in-the-loop model is mandatory for external high-impact operations.

Four Practical Lessons for AI Agent Developers

This case serves as a wake-up call for the rapidly developing AI Agent field. At least four lessons deserve serious attention from every practitioner:

Internal trial-and-error must be clearly distinguished from external impact. AI making mistakes internally is an acceptable learning process, but any outward-facing action requires human oversight and review. In practice, this means developers need to establish a clear "action classification" system within the Agent architecture — marking all operations that could reach external systems or people as high-risk, and mandating human confirmation before execution.
Physical-world common sense remains a fatal weakness of LLMs. You can't cook eggs without a stove, and eggs explode in ovens — this kind of seemingly simple common sense remains an enormous challenge for current large language models. Possible paths to solving this include: equipping Agents with structured environmental constraint files (such as equipment inventories, spatial parameters, daily consumption baselines), and introducing multimodal perception capabilities so AI can "see" the physical environment it serves.
Transparency is a non-negotiable bottom line. If external parties (suppliers, government agencies) are interacting with an AI, they have the right to know — this is a fundamental right to be informed. The EU AI Act already explicitly requires that when an AI system interacts with humans, it must inform the other party that they are interacting with an AI system, unless this is already obvious from the context of use.
"Entertaining" does not equal "ethical." The Hall of Shame story has great viral potential, but viral appeal should not obscure the ethical flaws in the experiment's design. Throughout AI's development history, a similar "move fast and break things" experimental culture has repeatedly triggered public trust crises — from early social media emotion manipulation experiments to unauthorized deployment of facial recognition technology, the lessons are profound.

The future of AI Agents is undoubtedly exciting, but the path to that future should not come at the cost of wasting innocent third parties' time and energy. Responsible experimental design is the true foundation for this field to earn public trust.