Coasty: API Infrastructure Built for Computer-Use Agents

What Are Computer-Use Agents

As large language models continue to grow more capable, the AI field is shifting from "conversational assistants" to "autonomous executors." Computer-use agents are AI systems that can operate computer interfaces the way humans do — clicking buttons, filling out forms, browsing the web, switching between applications, and completing complex multi-step tasks.

This technology didn't emerge from thin air. It sits at the intersection of three technical threads: computer vision, reinforcement learning, and large language models. Early experiments date back to reinforcement learning research in the 2010s, such as DeepMind's ATARI game agents, which learned to control interfaces through raw pixel input. But the real turning point that made this direction practical was the rise of multimodal large models — GPT-4V, Claude 3, and others developed the ability to "read a screenshot and reason about the next action," unifying perception and planning in a single system. Anthropic's release of the Computer Use API in 2024 marked the transition from research prototype to engineering-ready technology, triggering a wave of rapid infrastructure development across the industry.

There's a simple but powerful logic behind all of this: rather than building custom API integrations for every piece of software, let AI interact directly with existing graphical interfaces. The vast majority of enterprise software and websites are designed for human operation — as long as AI can "see the screen and control the mouse and keyboard," it can theoretically automate almost any interface-based workflow.

Coasty (YC S26 batch), which recently made its debut on Hacker News, is an infrastructure project aimed squarely at this space. It provides a suite of APIs designed specifically for computer-use agents, focused on solving the low-level engineering challenges developers face when building these kinds of applications.

What Problem Coasty Is Trying to Solve

Building a computer-use agent sounds straightforward, but the engineering challenges pile up quickly in practice. Developers need not only a stable virtual environment (typically a cloud-hosted browser or desktop instance), but also must handle the tedious details of screenshot capture, mouse and keyboard control, state synchronization, and session management.

The "execution layer" Coasty provides typically consists of several technical components: isolated cloud-based virtual environments (commonly using container-based approaches like Docker or lightweight VMs like Firecracker — the same technology underlying AWS Lambda) to ensure each agent session is isolated and controlled; a screen capture and streaming pipeline that balances low latency against bandwidth consumption, with common solutions including VNC protocol or WebRTC-based real-time transport; and an input control layer that translates the model's output — "click coordinates," "keyboard input," and similar instructions — into system-level operations. The complexity of this architecture is comparable to building a small cloud desktop service, which explains why the cost of building it from scratch is so high.

Every team rebuilding this infrastructure from the ground up wastes time and money on duplicated effort. Coasty's core value proposition is abstracting these low-level capabilities into a unified API interface, so developers can focus on the agent's decision-making logic rather than setting up and maintaining environments.

From a positioning standpoint, Coasty sits in the "AI infrastructure layer" — it doesn't serve end users directly, but rather the developers and companies building automation products, AI assistants, or next-generation RPA (Robotic Process Automation) tools. These kinds of platforms are the utilities of the agent era: as agent applications scale, demand for this infrastructure will scale with them.

It's worth noting that the new generation of computer-use agents is fundamentally different from traditional RPA. Legacy RPA tools (represented by UiPath and Automation Anywhere) rely on pre-recorded action scripts or fixed UI element locators (like XPath or CSS selectors) — any change to the interface breaks the workflow, leading to extremely high maintenance costs. The breakthrough with new-generation agents is the introduction of semantic understanding. Instead of looking for "the button in the third row, second column," the agent understands "where is the button to submit the order" — giving it a meaningful ability to adapt to interface changes. This is the core reason the industry views these agents as a disruptive replacement for traditional RPA.

The Technical Significance of API-ifying Computer Control

Wrapping computer-use capabilities in an API is a technical trend worth watching closely. It means that "operating a computer" can be triggered and composed programmatically, like calling a function.

Lowering the Development Barrier

For developers, the most immediate value of API-ification is a lower barrier to entry. In the past, building an agent that could autonomously complete web-based operations required familiarity with browser automation tools like Playwright or Selenium, plus the ability to handle anti-scraping measures, session persistence, error recovery, and more. With a standardized API, this complexity is encapsulated server-side, and developers only need to focus on "what they want the agent to do."

Complementing Mainstream Model Capabilities

Understanding this complementary relationship requires clarifying the "two-layer architecture" of current AI agents. The model layer (e.g., Anthropic Computer Use, OpenAI's CUA model) handles visual perception and decision-making: it receives screenshots, interprets the current state, and outputs the next action. The execution layer is responsible for carrying out those instructions: maintaining the virtual environment, executing mouse and keyboard actions, capturing new screenshots, and feeding them back to the model. The two layers communicate via a standardized action space protocol with primitives like Screenshot, Click(x,y), and Type(text).

OpenAI, Anthropic, and others have already released models with computer-use capabilities (such as Anthropic Computer Use and OpenAI Operator). But these models only handle "decisions" — they output instructions like "click here" or "type this text." The actual execution still requires a reliable runtime environment. APIs like Coasty fill this execution-layer gap, forming a complementary relationship with the model layer. This decoupled design lets developers freely mix and match different models with different execution environments — and Coasty is betting that the execution layer will become an independent value node in the stack.

Scalability and Concurrency Support

API-ification also enables scale. Enterprise applications often need to run hundreds or thousands of agent instances simultaneously. Cloud-based APIs are naturally suited for elastic scaling and concurrent session management — an advantage that local solutions simply can't match.

Market Competition and Real-World Challenges

The infrastructure space for computer-use agents is already quite crowded. Browserbase focuses on browser automation, offering hosted browser services built on Playwright. E2B provides broader code execution sandboxes, positioning itself as a general-purpose runtime for AI agents. Anthropic's official sandbox is deeply integrated with its Computer Use API, giving it the natural advantage of co-optimized model and execution environments. In this landscape, differentiation for new entrants typically comes from three dimensions: depth in a specific vertical (such as operating enterprise intranet software), extreme price-performance optimization, and developer experience (SDK quality, debugging tools, observability). As an early-stage YC S26 project, Coasty will need to find clear differentiation in stability, latency, pricing, and developer experience.

Beyond competition, this category of technology faces several common challenges:

• Reliability: The error tolerance for UI-based operations is far lower than for API calls. A single popup or page redesign can break an agent mid-task — robustness is the core technical problem.
• Cost control: The compute overhead of running virtual environments and vision models is significant. Striking the right balance between performance and price directly affects commercial viability.
• Security and compliance: Having AI autonomously operate computers involves sensitive data and permission management. Enterprise customers have extremely strict security requirements.

Where Agent Infrastructure Is Headed

Coasty's emergence is a microcosm of the AI agent ecosystem maturing. Once model reasoning capabilities are strong enough, the industry's focus shifts from "the model itself" to "how to make the model act reliably." A whole new infrastructure stack is taking shape around agent execution environments, tool calling, memory storage, and orchestration.

For developers and founders, this represents both opportunity and a cautionary note: pure API wrappers are easy to replicate. Real moats typically come from deep investment in reliability, cost efficiency, and specific use cases. Competition in execution-layer infrastructure is fundamentally a long-term contest of engineering capability and operational efficiency. YC's backing gives Coasty early resources and credibility, but whether it can stand out in an intensely competitive field remains to be proven by time and the market. What's certain is that the proliferation of tools like this is accelerating AI's evolution from "eloquent" to "effective."

Coasty's discussion on Hacker News is still in its early stages. Interested developers can continue to follow its technical documentation and product updates.

Key Takeaways