GitHub Copilot Custom Agents: From Local to CI at Scale

GitHub Copilot's capabilities are expanding beyond "a completion tool inside the IDE" into "an orchestration-ready, distributable, observable agent platform." In a talk at AI Engineer, GitHub's Jose Palafox systematically broke down the three-tier rollout path for custom agents: local creation, org-wide distribution, and scaling into CI pipelines. This isn't a feature list—it's an engineering methodology for "how to actually get agents running in a team."

Understanding Copilot's Underlying Data Flow

No matter which entry point you use—IDE, GitHub CLI, SDK, or mobile—requests ultimately converge on the same core: the Copilot API Service, or the so-called "harness." This layer handles infrastructure work like chunking input, deduplication, authentication, metrics recording, and audit logging, then forwards requests to model providers and filters the results before displaying them to the user.

Once you understand this architecture, it's clear why "switching entry points" doesn't change an agent's behavioral logic. The CLI supports the slash remote command, letting you close your laptop and keep controlling the same session from your phone; the SDK lets you embed agents into Slack or Teams to expose functionality to specific teams. Entry points are the doors; the harness is the room.

Harness, in the AI engineering context, refers to an orchestration and governance layer sitting between the application and the underlying models. It typically doesn't handle model inference itself; instead it manages the pre- and post-processing of requests: chunking and deduplicating input, authentication and permission checks, collecting call metrics, persisting audit logs, and routing requests to different model providers with a unified response format. Thinking of the harness as "the model's operating system" is apt—upper-layer applications (IDE, CLI, SDK) only face one interface, while lower-layer models can be swapped or mixed at any time. This also explains why Copilot's multiple entry points don't cause behavioral fragmentation: all entry points share the same harness, meaning authentication, quotas, logging, and model selection policies are centrally controlled. Adding an entry point just adds another frontend without changing the runtime semantics of agents. Understanding the boundaries of the harness helps you judge which capabilities can be reused and which must be reimplemented at the upper layer.

Local Agents: From Built-in to Custom

The CLI ships with a set of built-in agents that are themselves excellent learning samples. For example, the explorer agent uses a small model like Haiku to explore the codebase, avoiding using large models for low-value tasks; the rubber duck agent uses GPT to review plans written by Opus, leveraging the training-data differences between models for cross-validation. Each agent explicitly declares the model and instructions it uses—precisely the two core tunables of a custom agent.

Creating a custom agent is lightweight: type the agent command, choose a scope (project-level, user-level, or org-level), and Copilot automatically handles the scaffolding—writing descriptions, formatting front matter, and structuring success criteria and methodology. You only need one natural-language description; the rest can be fine-tuned manually.

Distribution: Marketplace and .GitHubPrivate

Once an agent is built, the next question is almost inevitably "how do I get others to use it too." Palafox's answer is a two-tier mechanism.

The first tier is the Marketplace: a repository with extra JSON configuration that provides lightweight package management—bulk updates, versioning, security vulnerability notifications, and broadcasting to downstream users. The second tier is the org-level .GitHubPrivate repository, where admins can push agent files to the entire organization's IDE and CLI environments so team members directly invoke the same agent.

There's an easily overlooked cost angle here: Palafox explicitly points out that the cheapest way to use this kind of product is to raise the cache hit rate—that is, making rounds on common data match as precisely as possible. Reusable agents let teams solve problems the same way rather than each reinventing the wheel, which both improves cache rates and lowers the overall bill. Standardization isn't just an engineering norm; it's also a cost strategy.

Cross-repository collaboration is achieved through the orchestrate command. Compliance-type agents typically can only act within their own repository, whereas orchestrate sets goals across a series of GitHub issues and launches sub-agents in target repositories, allowing microservice API changes and downstream modifications to proceed in parallel.

Cache hit rate here refers to the prompt caching mechanism of large model APIs. Most model providers cache repeated prefix content in requests (such as system instructions, long context, and tool definitions), and the cached portion is billed at a significantly lower price, often with lower latency. So if a team uses agents with identical structure and consistent instructions, a large number of request prefixes will reliably hit the cache; conversely, if everyone rewrites instructions and assembles context in their own way, the cache hit rate drops sharply and the bill rises accordingly. Palafox links "reusable agents" with "cost optimization" for exactly this reason: standardized agents not only unify how work gets done but also make request prefixes converge, turning cache hits from occasional to structural. When evaluating the cost of an agent platform, cache hit rate should be considered alongside per-token price and call volume.

CI Pipelines: Scaling Agentic Workflows

Local agents have inherent limits: they're tied to your device and session, and if the device goes offline or the network drops, the workflow breaks. More importantly, when everyone uses AI on their own machines, the organization sees no runtime data at all—it can't tell whether an agent is overly verbose, whether tool calls succeed, or whether reasoning is being misused.

Agentic Workflows move the workload into CI. The core mechanism is simple: the Copilot CLI supports headless mode, passing a one-shot prompt via -p, and placing it in a GitHub Actions runner or any CI system lets it be triggered by repository events—updating docs on release, refreshing context files weekly, or scanning the codebase on a schedule.

Palafox gave a very practical scenario: agent-driven development tends to produce lots of duplicate code and poor namespace management, so you can set up a scheduled agent to scan the codebase, identify deduplication opportunities, and generate a series of issues. The framework includes built-in human-in-the-loop trigger points, where developers use slash commands in issue comments to decide whether to advance to the next stage—planning, PRD generation, implementation, acceptance—each step gated by human review and cost control.

Headless mode means a CLI tool runs without an interactive terminal, not waiting for user input but receiving the complete prompt in one shot via parameters (like -p), then outputting results and exiting. This is the key prerequisite for moving local agents into CI: a GitHub Actions runner has no human sitting in front of it and can't respond to slash commands or confirmation dialogs, so only headless invocation can be triggered by repository events (push, release, schedule). Headless mode usually also requires injecting authentication via environment variables or secrets, and agreeing on exit codes and structured output formats so the pipeline can judge success. The trade-off is losing the opportunity for interactive correction, so human-in-the-loop must move to asynchronous channels like issue comments and PR reviews, where people decide whether to advance to the next stage.

A Fully Agent-Driven Reference Sample

GitHub's own Agentic Workflows project is itself the largest-scale validation: about 200 agents run at different stages of the pipeline, and the team has even set an internal goal of "developing only on mobile devices," interacting with the application entirely through slash commands. One meta-agent checks the daily reports of the other agents every day, generating a markdown report as input for the next agent.

The Agentics project under the GitHub Next organization provides about 30 out-of-the-box pipeline agents covering scenarios like code deduplication and QA review, which you can pull directly into your own pipeline.

Observability Is the Prerequisite for Scale

The biggest benefit of moving agents into CI is that every agent is wired into OTEL instrumentation. You can send the data to a SIEM or logging platform, pivot by average token usage, and observe how a deduplication agent performs and costs across repositories; you can set rules that automatically trigger an investigation when an agent exceeds the P90 cost threshold; and when tool calls fail repeatedly, you can A/B test different front matter or even different models, ultimately switching to a cheaper option.

The takeaway is clear: scaling agents isn't about "getting more people to use AI" but about "letting the organization see, govern, and optimize the operation of every agent." From local to CI is essentially a shift from a personal productivity tool to a governable engineering asset.

OTEL is short for OpenTelemetry, an open-source, vendor-neutral observability standard that unifies the collection and export protocols for three types of telemetry data: traces, metrics, and logs. For agents, OTEL instrumentation can break a task into traceable spans: model call latency, token usage, tool calls and their success or failure, retry counts, and so on, then export them to a SIEM, logging platform, or APM system. With a unified schema, organizations can make cross-repository and cross-agent comparisons—for example, pivoting cost by token usage, setting P90 threshold alerts, or A/B testing repeatedly failing tool calls. Without this instrumentation layer, agent operation is a black box, and scaling can only rely on intuition rather than data.