Pi AutoResearch: AI Runs Experiments Automatically, No More Manual Trial-and-Error for Performance Optimization

The Pain of Performance Optimization: One Line of Code, An Entire Afternoon Wasted

Every developer who's done performance optimization knows the drill: change one line of code, run a test, check the numbers, then decide whether to keep it or roll back. Back and forth, and suddenly the afternoon is gone. What's worse, you might try 20 different approaches only to discover that the third one was optimal all along—making all those other rounds of experimentation a complete waste.

This repetitive "experiment → measure → decide" loop is fundamentally a process that can be automated. Pi AutoResearch was built to solve exactly this problem.

What Is Pi AutoResearch? The Core Idea in One Sentence

Pi AutoResearch is an extension plugin for the Pi programming assistant, inspired by Andrej Karpathy's Auto Research project (which has already earned 1,700+ Stars on GitHub). Its core logic is refreshingly straightforward:

Think of an approach → Run it → Measure it → Keep the good, discard the bad → Loop infinitely

In plain terms, it fully automates the mental model humans use when doing performance optimization. AI is no longer just "giving you suggestions"—it's an independent researcher capable of completing the entire experimental loop on its own: proposing hypotheses, validating them, and iterating.

Workflow Breakdown: From Launch to Results

Start an Experiment with a Single Sentence

The user experience is incredibly smooth. You only need to tell it two things:

1. What metric to optimize: e.g., build speed, bundle size, page load time
2. What command to run: e.g., pnpm build

Everything else is handled automatically by the AI.

Five Steps of the Automated Experiment Loop

Pi AutoResearch's complete workflow includes the following stages:

• Auto-generate optimization plans: AI proposes specific optimization strategies based on the target metric and code context
• Auto-modify code: Applies the optimization directly in the project without manual intervention
• Auto-run tests: Executes the specified build or test commands and collects performance data
• Auto-record results: Each experiment round is automatically committed to Git, with results written to log files
• Auto-decide and iterate: Compares before-and-after data, keeps superior approaches, rolls back regressions, then moves to the next round

Resume from Breakpoints: No Fear of Interruptions

A very practical design detail: even if the context is reset (e.g., closing the editor or session timeout), a new AI instance can recover progress from the log files and continue running experiments. You can confidently let it run in the background for extended periods without worrying about losing progress.

Use Cases: Which Optimization Tasks Can You Hand Off?

Pi AutoResearch is suitable for virtually any optimization goal that can be measured numerically:

• Build optimization: Webpack/Vite build speed, bundle size reduction
• Test optimization: Execution time for unit tests and integration tests
• Model training: Training loss reduction
• Frontend performance: Lighthouse^Ad scores, first contentful paint, and other page performance metrics

The editor also includes a built-in real-time dashboard where you can see how many rounds have been completed and what the current best value is at any time—keeping the entire process transparent and controllable.

Why This Approach Deserves Attention

From "AI-Assisted Coding" to "AI-Autonomous Experimentation"

Traditional AI programming assistants (Copilot, Cursor, etc.) are primarily "suggestion-based"—they give you code snippets, and you decide whether to adopt them. Pi AutoResearch represents a more advanced paradigm: AI autonomously completing the full experiment cycle.

This model is particularly well-suited for performance optimization for three reasons:

1. Clear evaluation criteria: Performance metrics are objective numbers that don't require human subjective judgment
2. Low per-experiment cost: Running a build or test takes only a few minutes, making trial-and-error cheap
3. Large search space: There could be hundreds or thousands of optimization combinations, making manual one-by-one attempts far too inefficient

A Real-World Implementation of Karpathy's Auto Research Philosophy

The inspiration behind this project—Karpathy's Auto Research—represents an important direction in AI-assisted research. The core philosophy is to have AI serve not merely as a tool, but as an agent capable of independently performing the "hypothesis → experiment → validation" scientific method loop.

Pi AutoResearch brings this philosophy from academic research into everyday development, dramatically lowering the barrier to entry. You don't need to build a complex experimental framework—just install a plugin, write a single instruction, and you're up and running.

Summary: AI Runs Experiments, Humans Make Decisions

The core value of Pi AutoResearch is liberating developers from repetitive performance optimization trial-and-error. You no longer need to manually try 20 approaches before picking the best one—let AI handle the "grunt work experiments" while you simply review the results.

For teams that frequently need performance tuning, this kind of automated experimentation tool can significantly boost efficiency. Of course, whether the final optimization plan is sound and free of side effects still requires the professional judgment of developers.

AI runs experiments, humans make decisions—this is perhaps the most pragmatic human-AI collaboration model for the current stage.