PaperOrchestra Open Source Project Explained: Automatically Generating Research Papers with Coding Agents

What is PaperOrchestra: An Open Source Implementation Based on Google's Paper

Writing research papers is one of the most time-consuming aspects of academic work. From literature review and experimental design to final drafts, a high-quality research paper often takes weeks or even months. PaperOrchestra is an open source project attempting to dramatically shorten this cycle with AI.

PaperOrchestra is a community open source reproduction of Google's research paper of the same name, employing an architecture that combines Skills Benchmarks and Autoraters to achieve automated research paper generation. The project has already garnered nearly 400 stars on GitHub, sparking considerable discussion among developers and researchers.

Google's PaperOrchestra paper, published in 2025, is part of its "AI for Science" research agenda. The paper's core contribution is proposing a systematic evaluation framework that transforms the highly unstructured task of paper writing into a quantifiable, benchmarkable set of sub-skills. This approach draws from the continuous integration philosophy in software engineering—ensuring overall output quality by decomposing complex systems into independently testable modules. The paper also introduces the concept of Autoraters, using LLMs themselves as evaluators to judge the quality of generated content, which aligns with the recent "LLM-as-a-Judge" research paradigm.

Core Design Philosophy: An Agent-Driven Architecture Without API Keys

No Dependency on Any API Keys or LLM SDKs

PaperOrchestra's most notable design decision is: absolutely no dependency on API keys or LLM SDKs. Users don't need to directly call interfaces from major model providers like OpenAI or Anthropic. Instead, the entire paper generation workflow is driven through existing Coding Agents.

Coding agents are a rapidly emerging category in the AI toolchain since 2024. Unlike traditional chat-based AI assistants, coding agents have the ability to autonomously execute multi-step tasks—they can read file systems, execute terminal commands, modify code, and verify results, forming a complete "perceive-decide-execute" loop. This capability means coding agents are no longer limited to code completion but can handle complex engineering orchestration tasks. PaperOrchestra leverages this characteristic, repositioning coding agents from "assistants that write code" to "autonomous agents that execute complex academic workflows."

Currently supported coding agents include:

• Claude Code: Anthropic's command-line coding assistant
• Cursor: A code editor with integrated AI capabilities
• Antigravity: An emerging AI coding tool
• Cline: An AI coding plugin for VS Code
• Aider: An open source AI pair programming tool

The direct benefit of this design is lowering the barrier to entry—as long as you have access to any of the above coding agents, you can run the entire paper generation workflow without needing to manage additional API keys and billing.

How Skills Benchmarks and Autoraters Work Together

PaperOrchestra's architecture draws from the core ideas of Google's original paper, decomposing paper writing into multiple quantifiable "skills" and then using autoraters to gatekeep output quality at each step.

Skills Benchmark

The system breaks down research paper writing into the following subtasks:

• Literature review and related work synthesis
• Formal articulation of research questions
• Experimental design
• Results analysis and discussion writing
• Overall paper structure organization

Each subtask corresponds to a set of measurable metrics, making the paper generation process no longer a black box.

Autoraters

Autoraters are responsible for scoring the output of each stage to ensure generated content meets basic academic standards. This "divide and conquer" strategy makes the entire system more controllable—when output from a particular stage falls short, you can debug and optimize that specific component without starting from scratch.

The technical foundation behind autoraters is the "LLM-as-a-Judge" paradigm, which uses large language models themselves to evaluate the output quality of another model. This method was first systematically proposed by UC Berkeley's LMSYS team in 2023 and has been widely applied in model alignment, content moderation, and automated testing. Its core advantage is scalability—compared to human review, LLM evaluation can be completed in seconds at extremely low cost. However, this approach also has known limitations, including position bias (tendency to prefer answers listed first), verbosity bias (tendency to give higher scores to longer responses), and self-preference (tendency to rate outputs similar to its own style more highly). In the highly specialized context of academic paper evaluation, these biases may be further amplified, making this an area where PaperOrchestra needs continuous optimization in practical applications.

Technical Implementation Details and Ecosystem Positioning

Python-Driven Orchestration Layer

The project is written in Python, consistent with the mainstream technology stack for AI research tools. From an architectural perspective, PaperOrchestra is essentially an Orchestration Layer—it doesn't contain a large language model itself but coordinates various coding agents to complete complex paper writing tasks.

The orchestration layer is a core design pattern in current AI application architecture, with frameworks like LangChain, CrewAI, and AutoGen all falling into this category. The core concept is decoupling "intelligence" from "process control"—the orchestration layer itself doesn't possess reasoning capabilities but is responsible for task decomposition, context management, tool invocation, and result aggregation. The advantage of this architecture lies in its high flexibility and composability, but it also introduces additional complexity, including context window management, error propagation in multi-step tasks, and inconsistent output formats across different models. PaperOrchestra's choice of this architecture is essentially a bet on the continued maturation of the coding agent ecosystem.

This architectural design has three clear advantages:

1. Model agnosticism: The underlying LLM can be switched at any time without being locked into a specific model
2. Cost flexibility: Users can choose coding agents at different price points based on their budget
3. Extensibility: New coding agents can be easily integrated into the system without modifying core logic

PaperOrchestra vs. Other AI Paper Writing Tools

There are already numerous AI-assisted paper writing products on the market, such as Elicit, Semantic Scholar's AI features, Jenni AI, and more.

The AI-assisted research tool market experienced explosive growth between 2023-2025. Elicit focuses on systematic literature reviews, helping researchers quickly filter relevant papers through semantic search and automatic data extraction; Semantic Scholar's AI features (TLDR, citation intent classification, etc.) focus on paper discovery and comprehension; tools like Jenni AI and Writefull specialize in academic writing polish and grammar optimization. Additionally, Consensus focuses on extracting research conclusions from papers, while SciSpace provides paper reading and explanation features. The common characteristic of these tools is their focus on a specific stage of the research workflow rather than end-to-end coverage.

PaperOrchestra differentiates itself in three ways:

Comparison Dimension	PaperOrchestra	Other AI Paper Tools
Coverage	End-to-end full-process automation	Usually focused on a single stage (e.g., literature search, polishing)
Open source level	Fully open source, auditable and improvable	Mostly closed-source commercial products
Driving mechanism	Coding agent-driven	Simple prompt engineering or specialized models

PaperOrchestra's ambition is to connect the complete chain from conception to final draft, but this also means it may not be as refined as specialized tools at any single stage. This end-to-end open source approach allows researchers to deeply understand the implementation details of each stage and provides a foundation for secondary development.

Use Cases and Current Limitations

Suitable Scenarios

PaperOrchestra is currently well-suited for the following purposes:

• Rapid prototype validation: Generate paper drafts before investing significant time to verify whether research ideas are viable
• Literature review acceleration: Automatically generate initial drafts of the related work section, reducing repetitive labor in early-stage research
• Paper writing education: Help students intuitively understand the structure and writing conventions of each paper section

Issues That Must Be Addressed

However, AI-automated paper writing also faces several unavoidable challenges:

• Academic integrity boundaries: How should originality be defined for automatically generated paper content? Policies across universities and journals remain inconsistent. The ethical boundaries of AI-generated academic content are evolving rapidly. In early 2023, top journals like Science and Nature explicitly prohibited listing AI as a paper author but allowed disclosure of AI-assisted use in methodology sections. By 2024, most major publishers (including Elsevier, Springer Nature, IEEE) have formed a relatively unified stance: AI can be used as a writing assistance tool, but authors must bear full responsibility for content accuracy and originality, and must clearly state how and to what extent AI was used in the paper. Funding agencies such as the National Science Foundation (NSF) and the European Research Council (ERC) have also begun incorporating AI usage provisions in their project application guidelines. This policy uncertainty is the biggest non-technical risk facing end-to-end paper generation tools like PaperOrchestra.

• Quality ceiling: Autoraters currently cannot truly replace the deep judgment of peer review

• Innovation bottleneck: AI excels at integrating and recombining existing knowledge, but its ability to produce genuinely original research insights remains limited

These issues involve not only the technical level but also deeper transformations in academic ethics and research paradigms.

Conclusion: The Future Direction of AI Paper Writing Tools

PaperOrchestra represents an important trend in AI-assisted research—evolving from single-point assistance tools to end-to-end automated systems. While fully replacing human researchers is still a long way off, the emergence of such projects is redefining what's possible in research workflows.

For researchers and developers, following and participating in open source projects like this serves both as a window into understanding AI's capability boundaries and as a starting point for thinking about the future of scientific research.

Project URL: GitHub - Ar9av/PaperOrchestra