AR Glasses + AI Coding in Action: A Vibe Coding Experience Where You Point and Speak to Modify Code

When AR Glasses Meet Vibe Coding

Can you imagine a programming scenario like this? Put on AR glasses, point your finger at an element on the screen, say "make this video bigger and move it to the right," and the AI automatically modifies the code and renders the result in real time. This isn't science fiction—it's the new interaction experience brought by Monoco Glass.

This demo video showcases an interaction model that disrupts traditional programming—using AR glasses combined with an AI Agent to complete code modifications through natural language and gesture commands. The entire process requires no keyboard input, no memorized syntax, truly achieving a "point and get" programming experience.

What is Vibe Coding?

Vibe Coding is a programming paradigm concept proposed by OpenAI co-founder Andrej Karpathy in early 2025. The core idea is that developers no longer write code line by line but instead describe their "feeling" and "intent" through natural language, letting AI handle the actual implementation. This concept quickly sparked industry-wide discussion because it touches on a fundamental shift in programming: from precise syntax expression to fuzzy intent communication. The rise of Vibe Coding is inseparable from breakthroughs in large language models for code generation—when models like GPT-4 and Claude can understand context and generate high-quality code, the role of human developers naturally migrates from "code writers" to "requirement expressers." The addition of AR glasses provides the most intuitive input method yet for this paradigm.

Practical Demo: From Web Layout Adjustments to 3D Scene Rendering

Natural Language Web Layout Adjustments

In the demo, the user first faces a web layout containing text and video. The user points at the video element and issues a voice command: "Make this video bigger, move it to the right, and fill the empty space."

After receiving the command, the AI Agent automatically analyzes the current page structure, understands the user's intent, and generates the corresponding code modifications. A few seconds later, the page finishes re-rendering. The video did get bigger and moved to the right, but it didn't completely fill all the white space as expected.

It's worth understanding how the AI Agent works here: it's not simply converting speech to text and searching for code snippets. It's an intelligent system capable of perceiving the environment, autonomously planning, and executing multi-step tasks. In this scenario, the AI Agent needs to complete a full perception-understanding-execution pipeline—using computer vision to identify the specific DOM element the gesture points to, combining voice commands to understand modification intent, analyzing the current code structure to generate a differential modification plan, and finally triggering the rendering engine to present results in real time. This process involves the collaborative work of multiple cutting-edge technologies including multimodal fusion (vision + voice), code understanding, and generation.

Iterative Optimization: Modifying Code Like a Conversation

This demonstrates a core advantage of Vibe Coding—iterative modifications feel as natural as everyday conversation. After noticing the result wasn't ideal, the user directly adds clarification: "I want the video centered on the right side, filling all the space to the right of the text."

The AI processes the command again, this time precisely understanding the meaning of "centered" and "fill the right side," generating more accurate layout code.

The final result fully meets expectations. The entire process is like conversing with an exceptionally perceptive designer, continuously refining requirements until satisfied.

Works for 3D Projects Too

Even more impressively, this interaction method isn't limited to 2D web layouts. The demo also shows manipulation of 3D projects—the user points at a group of colorful blocks and says "make all these boxes a uniform yellow," and the AI immediately completes the color modification of all objects in the 3D scene.

From colorful blocks to uniform yellow, done with a single sentence. To understand the technical sophistication behind this, you need to know that 3D scenes are typically built with graphics libraries like Three.js—traditionally, modifying the material color of a 3D object requires developers to find the corresponding Mesh object, access its material property, modify the color value, and trigger a render update, a process with a considerable learning curve for non-professional developers. The AI's ability to precisely map the natural language instruction "turn yellow" to Three.js material modification code demonstrates the maturity of large language models in understanding specific framework APIs. This means whether it's front-end development, UI design, or 3D modeling, this interaction paradigm has broad applicability.

Technical Considerations: Why AR is the Ideal Vehicle for Vibe Coding

AR (Augmented Reality) glasses overlay digital information onto the real-world view. Unlike VR, which completely isolates users from the physical world, AR preserves awareness of the real environment. The current consumer AR glasses market is at a critical pre-explosion stage: Meta Ray-Ban smart glasses focus on lightweight design, Apple Vision Pro uses spatial computing as its selling point, while developer-focused products like Monoco Glass explore more vertical productivity applications.

Traditional AI programming assistants (like Claude Code, Cursor, etc.) still require users to input prompts in a terminal or IDE. AR glasses bring two key breakthroughs:

1. Spatial Pointing: Users can directly "point" at the element they want to modify, eliminating the linguistic cost of describing target positions. This is precisely the unique value of AR glasses in programming scenarios—the user's gaze and gestures naturally carry spatial coordinate information, a dimension that traditional keyboard-mouse interaction cannot provide.
2. Immersive Feedback: Modification results appear in real time within the field of view, forming a tight "command-feedback" loop.

This interaction method lowers the barrier to programming to unprecedented levels. You don't need to know CSS flex layout syntax or understand Three.js material properties—you just need to know what effect you want.

Current Limitations and Future Directions

The demo also reveals that AI doesn't perfectly understand user intent on the first try. After the first command execution, the video got bigger and moved to the right, but didn't precisely "fill" the space. Behind this lies natural language ambiguity—a long-standing challenge in human-computer interaction. Commands like "fill the empty space" rely on extensive shared visual common sense when communicated between humans, but AI systems need to translate them into precise CSS property values or pixel coordinates. Current mainstream solutions include multi-turn dialogue to clarify intent, incorporating visual understanding models to analyze the current layout state, and building user preference memory to reduce ambiguity.

The Vibe Coding paradigm effectively shifts the cost of "eliminating ambiguity" from "users learning precise syntax" to "AI iterating through multiple rounds of understanding"—which is precisely the philosophy of Vibe Coding. It doesn't pursue perfection on the first try but instead approaches the target through rapid iteration. When the cost of modifying code drops to "saying one sentence," trying a few more times is simply not a problem.

What This Means for Developers

The emergence of tools like this doesn't mean programmers will lose their jobs—rather, the definition of programming is being rewritten. Future developers may be more like "AI programming commanders"—guiding AI to complete specific implementations with high-level intent and aesthetic judgment.

Keyboards won't actually be thrown away, but their proportion in the programming workflow is indeed declining rapidly. When the combination of AR glasses + AI Agent becomes sufficiently mature, the very meaning of "writing code" may be fundamentally different from what it is today.

Key Takeaways

• Monoco Glass enables natural language programming through gesture + voice via AR glasses, allowing users to issue modification commands by pointing at elements
• Vibe Coding, proposed by Andrej Karpathy, replaces precise syntax with intent expression, with the AI Agent handling the multi-step perception-understanding-execution pipeline
• Supports real-time code modification for both 2D web layouts and 3D projects (like Three.js scenes), covering scenarios from front-end development to 3D modeling
• AR glasses provide spatial pointing and immersive feedback, shifting the cost of "eliminating ambiguity" from learning syntax to AI-driven iterative understanding