As AI Writes More Code, How Do We Govern It Safely? Two Open Source Projects Point the Way

When AI Becomes the Primary Contributor to Your Codebase

AI coding agents are evolving from assistive tools into a central force in the software development ecosystem. They're no longer just completing a few lines of code — they're writing the majority of code, modifying repositories, calling tools, accessing systems, making decisions, and potentially coordinating with other agents.

This leads to a counterintuitive conclusion: as AI takes on more coding work, human-in-the-loop involvement doesn't become optional — it may actually become more critical. The challenge is that the people responsible for reviewing, guiding, and governing these AI development agents themselves need new tools and new capabilities built for the AI era. And this point is severely underaddressed in current industry discussions.

This is the core argument that the Ikarus AI Safety Tech team posted on Reddit: AI will build more of our software, and humans must get better at governing it.

SafeAI: Understand an Agent Before It Enters Your Workflow

The first project, SafeAI, is positioned as a "static AI agent security and capability analyzer." Its core idea is straightforward: before an AI agent becomes part of a development workflow, we should be able to understand the boundaries of its capabilities.

Specifically, SafeAI tries to answer questions like:

• What capabilities does this agent have?
• What tools can it call?
• How high is its autonomy?
• What memory and identity does it possess?
• What potential risks might it introduce?

This "analyze first, integrate later" approach is essentially moving the security review process of traditional software supply chains upstream to the AI agent layer. Once an agent has permissions to access systems and execute actions, it's no longer just a piece of code — it's an entity with behavioral capabilities that should be evaluated before entering a production environment.

The "static analysis" qualifier is worth highlighting. Static Analysis refers to examining a program without actually running it, as opposed to dynamic analysis (which requires executing code to observe behavior). Applying static analysis to AI agents means inferring potential behavior and risk boundaries by inspecting metadata such as the agent's configuration files, System Prompts, Tool Definitions, and permission declarations — without actually running the agent. The advantage of this approach is that it's low-risk, repeatable, and automatable in CI/CD pipelines. The limitation is that an AI agent's actual behavior is often highly dependent on runtime context, and static analysis cannot fully cover all emergent behavior. SafeAI is therefore positioned more like a code linter or SAST (Static Application Security Testing) tool — a first line of defense, not a complete security guarantee.

OpenPulse: A "Early Warning Radar" for Open Source Dependencies

The second project, OpenPulse, focuses on a different long-standing but often overlooked pain point — dependency intelligence for the open source software ecosystem.

Modern software often depends on thousands of external projects, all of which are constantly changing: they get updated, deprecated, reach end-of-life (EOL), expose vulnerabilities, change maintainers, or produce other signals that eventually ripple into your own system.

What OpenPulse aims to do is help teams see these changes before they turn into "surprises." In other words, it transforms reactive "fire-fighting after the fact" into proactive "early warning before it happens." For modern engineering teams with large dependency footprints, this kind of dependency intelligence can significantly reduce supply chain uncertainty.

Software Supply Chain Security has become a major industry focus in recent years. The 2020 SolarWinds incident and the 2021 Log4Shell vulnerability made the entire industry realize that attackers can compromise downstream systems at scale by poisoning upstream dependencies. In this context, "Dependency Intelligence" goes beyond tracking known vulnerabilities (CVEs) — it extends to softer signals: whether a project has entered an unmaintained state, whether core maintainers have changed (maintainer account takeover is an emerging attack surface), whether licenses have quietly changed, and more. EOL (End of Life) refers to the point at which software officially stops receiving security updates and technical support — using EOL dependencies means known vulnerabilities will never be patched. What OpenPulse does is aggregate these signals scattered across GitHub, npm, PyPI, OSV databases, and other sources, transforming them into actionable early warnings for engineering teams.

The Shared Proposition Behind Both Projects

On the surface, SafeAI and OpenPulse address two different problems — "AI agent security" and "open source dependency governance" — but they point toward the same larger exploration:

AI will build more of our software, and humans must get better at governing it.

One project evaluates intelligent agents entering the system; the other monitors the external world the system depends on. Together, they form the early outline of a governance framework for the new paradigm of AI-driven software development. This combined approach is worth paying attention to, because it shifts the conversation from "how well can AI write code" to "how do we safely manage the code AI writes" — and the latter is usually where engineering organizations actually struggle.

"Human-in-the-Loop" (HITL) in AI system design specifically refers to mechanisms that retain human approval or intervention at critical decision points. It was originally widely used in machine learning model training (manual annotation, reinforcement feedback) and high-risk automation scenarios (such as autonomous driving and medical diagnosis). As AI agents gain the ability to directly manipulate code repositories, execute terminal commands, and call external APIs, the meaning of HITL is being redefined in the software engineering context — it's no longer just "approve a model prediction once," but involves access control policies, change review processes, anomaly detection circuit breakers, and other systemic design concerns. SafeAI and OpenPulse can be understood as two different dimensions of HITL infrastructure: the former provides decision-making support at the "introducing a new agent" checkpoint, while the latter continuously provides intelligence along the "dependency lifecycle" dimension. Together, they aim to transform human intervention from passive response into informed, proactive governance.

Recruiting "Founding Advocates": They Want Critics, Not Cheerleaders

The team admits they don't want to build in isolation and are looking for the first batch of "Founding Advocates." The target audience includes developers, security engineers, AI builders, open source maintainers, DevOps/SRE practitioners, architects, and anyone genuinely curious about these problems.

What's interesting is their attitude: they don't want blind promotion — they actively welcome pushback.

Tell us where we're wrong. Try the tools, break our assumptions, propose capabilities we haven't thought of, and tell us what would make these tools actually useful in real engineering organizations.

This kind of openness isn't rare in the open source community, but explicitly putting "challenging assumptions" first signals that the project is still in early shaping stages and welcomes outside perspectives to help define what responsible AI-driven software development should look like.

Interested developers can check out both projects on GitHub:

• SafeAI: ikaruscareer/SafeAI
• OpenPulse: ikaruscareer/OpenPulse

Final Thoughts

Both projects are still at a very early stage — more of an open proposition and invitation than a mature solution. But the questions they raise are spot-on: as AI agents become more deeply embedded in software development workflows, the governance capability gap is widening.

Whether or not SafeAI and OpenPulse ultimately become mainstream tools, the thinking they represent — providing AI-era software governors with matching intelligent tooling — is a direction the entire community should take seriously.