Gemini 3 Flash vs. CAPTCHA Visual Puzzles: Where Are the Limits of Multimodal Agent Capabilities?

When VLMs Meet Fun Visual Puzzles: A Down-to-Earth AI Test

Recently, a developer shared a compelling set of experiments on Reddit: using Google's latest Gemini 3 Flash Vision-Language Model (VLM) to tackle various CAPTCHA-style interactive puzzles on neal.fun.

Vision-Language Models (VLMs) represent a major breakthrough in multimodal AI in recent years. Unlike traditional single-modality models, VLMs can process both image and text information simultaneously, establishing semantic connections between the two. The core technical approach involves converting images into feature representations through a visual encoder (such as ViT), then fusing them with text tokens in a unified representation space, and finally performing reasoning through a large language model. Notable examples include OpenAI's GPT-4V, Google's Gemini series, and Anthropic's Claude 3. The capability boundaries of these models are expanding from simple image captioning to complex visual reasoning, spatial understanding, and multi-step task planning.

These puzzles are known for being tricky and creatively challenging — they've long been classic material for testing human logic and visual comprehension. CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) was originally designed to distinguish humans from machines. The interactive puzzles on the neal.fun website push this concept toward the playful and complex, incorporating spatial reasoning, physics simulation, logical sequencing, and other multidimensional challenges. These puzzles have unique value for AI evaluation: they're non-standardized and difficult to solve by memorizing training data; they require a combination of capabilities — visual perception, logical reasoning, and common-sense understanding; and their objectives are clear but the paths to solutions are open-ended, providing a genuine reflection of a model's generalization ability. Compared to traditional benchmarks like ImageNet classification or VQA, these open-ended interactive tasks more closely resemble real-world application scenarios. And now, they've become an excellent litmus test for evaluating the true capabilities of multimodal large models.

The developer's technical approach is noteworthy: using Playwright (a browser automation tool) to drive web interactions, combined with a custom execution framework (harness), enabling Gemini 3 Flash to "see" page content, understand puzzle objectives, and output corresponding action commands.

Playwright is a next-generation browser automation framework developed by Microsoft, supporting three major browser engines: Chromium, Firefox, and WebKit. Compared to the veteran tool Selenium, Playwright offers a more modern API design, more reliable waiting mechanisms, and more powerful debugging capabilities. Its core advantages include automatic waiting for elements to become interactable, built-in network interception and mocking, screenshots and video recording, and cross-browser compatibility. In AI Agent development, Playwright serves as the "hands and eyes" — capturing visual input through screenshots and executing AI decisions through simulated clicks, inputs, drag-and-drop actions, and more.

This combination essentially builds a simple visual Agent — one that not only needs to interpret images but also plan actions and interact with a real web environment.

A visual Agent refers to an intelligent agent system capable of perceiving visual environments and autonomously executing tasks. Its typical architecture includes three core modules: a perception module (capturing environmental state through screenshots, OCR, etc.), a decision module (the VLM understanding task objectives and planning action sequences), and an execution module (translating abstract actions into concrete operations like coordinate clicks). Unlike traditional RPA scripts, visual Agents don't rely on predefined DOM selectors or fixed workflows — instead, they adapt to page changes through visual understanding. Current technical challenges include coordinate mapping precision, error accumulation in multi-step reasoning, and state tracking in dynamic environments.

From "Understanding" to "Doing"