Hands-On with the CodeWorld Plugin: An AI Photo-Editing Tool Where You Point and It Changes

From Tedious Editing to "Point and Change"

In daily life, we often run into photo-editing needs: a stranger who wandered into a travel shot, clutter on a table, a background element we want to swap out... Traditionally, these tasks either required professional Photoshop skills or repeated trial and error with masks, brushing, and fill tools across various photo-editing apps—the barrier to entry wasn't low.

Recently, a Codex-native plugin shared by a creator on Bilibili—CodeWorld—offers a whole new approach: annotate directly on a visual canvas where you want to make changes, and leave the rest to the AI. This "what you see is what you edit" interaction makes photo editing accessible even to complete beginners.

In short, CodeWorld is a Codex-native infinite canvas plugin, and it's also an open-source project. It provides a visual canvas dedicated to conceiving, annotating, and generating images, combining AI image-editing capabilities with intuitive visual operations.

Further Reading: What Are Codex and Native Plugins

OpenAI Codex is a programming and agent tool built on large language models. It can understand natural-language instructions and invoke various tools to complete tasks. A "native plugin" refers to a feature extension deeply integrated into the Codex runtime environment—users can invoke capabilities like the canvas or editor directly within the conversation flow, without switching to other software. This design differs fundamentally from traditional standalone apps: it embeds tool capabilities into the AI conversation context, where the AI is both the executor of operations and the interpreter of instructions, creating a seamless "chat while you work" experience.

About the Infinite Canvas

An infinite canvas is a concept of a visual workspace unbound by fixed borders, where users can freely zoom, pan, and infinitely expand to place images, text, and annotation elements. It was first widely adopted in design and collaboration tools like Figma, Miro, and tldraw, and in recent years has increasingly combined with AI capabilities to become an important interaction medium for AI-native applications. Compared to traditional form-based or conversational interfaces, the infinite canvas can carry richer spatial information, letting users organize content in a way that's closer to human thinking—which is exactly why it was brought into the field of AI image editing.

CodeWorld Hands-On Demo: Editing Three Spots in One Photo at Once

Based on the creator's demo, the whole workflow is very intuitive. First, you type "Open the CodeWorld canvas for the current project" into the prompt, and the canvas opens right away. Next, you drag the image you want to edit directly onto the canvas, and you can start annotating.

The annotation method is quite interesting—instead of the traditional box-select or brushing, you draw a leader line with an arrow pointing to the spot you want to change, then type your editing instruction next to it. In the demo, the creator annotated three spots in one go:

• Pointing to the glasses on the coffee table, typed "remove the glasses";
• Pointing to the cat's head, typed "put a red Christmas hat on the cat's head";
• Pointing to the fruit on the plate, typed "change the fruit on the plate to a turkey."

Once the annotations are done, select the image, click "Edit by Annotations," then click Send. The plugin packages these annotations and prompts together and sends them to the AI for processing.

You may not have noticed, but this "leader line + text" annotation method essentially spatializes the user's intent—the AI not only knows what you want to change, but also clearly knows where in the image to change it. Compared to pure text prompts (like "remove the glasses"), instructions carrying coordinate information significantly reduce the chance of the AI "editing the wrong spot."

How Well Does It Edit: Only the Annotated Areas Change, Overall Details Preserved

After some processing by the AI, the new image was successfully generated. Looking at the results: the glasses on the coffee table disappeared, the cat is now wearing a red Christmas hat, and the fruit on the plate turned into a turkey—with natural transitions overall and no obvious signs of photo manipulation.

Comparing the before-and-after images reveals a key advantage: it only modified the annotated areas, while preserving the overall details of the rest of the image. This is precisely the hardest thing to get right in current AI image editing—many AI photo-editing tools, after making local edits, tend to alter the lighting, color tone, or even the appearance of unrelated objects across the entire image, causing a "pull one hair and the whole body moves" effect. By leveraging precise spatial annotations, CodeWorld does a good job of constraining the scope of edits, achieving controllable editing that hits exactly where you point.

This ability to "change only the local area without touching the whole" typically relies on Diffusion Models and inpainting techniques. Diffusion models generate images from random noise through progressive denoising, and this is the core principle behind mainstream generative models like Stable Diffusion, DALL·E, and FLUX. Inpainting techniques allow the model to regenerate content only within a specified mask region while referencing surrounding pixels to maintain the continuity of lighting and color tone. CodeWorld's "leader line + text" annotations essentially help the underlying model determine this mask region and semantic intent, thereby constraining the scope of edits—which also explains why it can achieve more precise local editing than pure text instructions.

For scenarios where you need to edit multiple areas at once, this batch-annotation, single-generation mode also greatly improves efficiency, saving the hassle of repeated back-and-forth edits.

Installation and Getting Started: Done in One Line

As a Codex-native plugin, CodeWorld's installation is quite simple. Users just need to copy the official installation content into Codex and press Enter, and Codex will automatically complete the installation and configuration of the CodeWorld canvas plugin—no manual environment fiddling required.

This "one-line, install-and-use" experience is also the direction pursued by today's MCP (Model Context Protocol) ecosystem and various AI Agent plugins—packaging complex tool capabilities into conversational invocations, so end users can gain powerful functionality almost effortlessly. MCP is an open protocol proposed by Anthropic in late 2024, aiming to establish a unified connection standard between AI models and external tools and data sources—much like a "USB interface" for the AI world. With such a standardized protocol, various plugins can connect to different AI clients at low cost and with high compatibility. CodeWorld's smooth "one-line, install-and-use" experience is built precisely on this ecosystem trend.

A Few Observations and Thoughts

CodeWorld's value lies not only in its ability to edit photos, but more in how it demonstrates an evolution of the AI interaction paradigm.

In the past, our interactions with AI image tools relied on pure text prompts, which required users to translate spatial intent into words—both laborious and prone to ambiguity. CodeWorld brings "visual canvas + spatial annotation" into the editing workflow, allowing human operational intent to be conveyed to the AI in a way that's closer to intuition. This division of labor—"humans point the direction, AI does the work"—is precisely a microcosm of human-machine collaboration in the age of Agents.

Of course, such tools still have their limits at present. It's heavily dependent on the capabilities of the underlying image-generation model. Whether it can reach professional-tool standards for extremely fine-grained commercial-grade retouching, complex structural reconstruction, and similar scenarios still needs to be validated across more real-world cases. At the same time, as an open-source project, its stability, generation consistency, and adaptability to different image types still await further polishing by the community.

But for everyday photo-editing needs of ordinary users, CodeWorld undoubtedly lowers the barrier. You don't need to understand layers or masks—as long as you can clearly state "where I want to change and what I want to change it to," the AI can handle the rest. This may well be the direction in which AI tools truly go mainstream: hiding professional capabilities behind simple interactions.

Key Takeaways