RoughCut: A Fully Automated AI Video Editing Tool Built with Codex

Why Build Your Own AI Video Editing Tool

For creators who publish videos frequently, editing is often the biggest bottleneck in content production. This Bilibili content creator admitted that opening video editing software was the "biggest obstacle" to publishing his videos—every operation relied on experience-based manual adjustments, and if you don't demand too much from the final product, about 80% of a typical video editor's features actually go unused.

He had long followed various high-quality open-source projects, but found them highly limited in real production settings: on one hand, his audience generally had limited hardware and couldn't run heavy models; on the other hand, many open-source projects were "watched once and then discarded," never truly integrating into a workflow. As Agent development tools like Codex and Claude Code matured, he decided to shift his focus toward productivity tools that "genuinely save time and are practically usable," integrating high-quality open-source capabilities into them.

Background on AI Agent development tools: Codex and Claude Code represent a new paradigm of AI-assisted programming. Codex is a code-generation model from OpenAI that can understand natural language instructions and translate them into runnable code; Claude Code is a command-line programming assistant from Anthropic based on the Claude model, supporting end-to-end execution of complex coding tasks in terminal environments. The maturity of such tools marks AI's evolution from "code completion" to "Agent development"—where AI can autonomously plan tasks, invoke tools, and iteratively fix errors to complete entire engineering projects spanning hundreds or even thousands of lines, rather than merely completing a few lines of code. Notably, there's a deep engineering reason behind this paradigm shift: traditional IDE plugin-style code completion (like early GitHub Copilot) relies on the user actively triggering it, whereas the Agent mode introduces a "Tool Use" mechanism that allows the AI to proactively read and write to the file system, execute terminal commands, and search codebases, forming a "perceive-plan-execute" loop. It is precisely because of this that a complete engineering project like RoughCut—encompassing frontend and backend, Docker configuration, and multi-model routing—could be independently "AI-generated" by a single creator without assembling a professional engineering team.

RoughCut (which the author calls "rough cut") is the first result of this line of thinking—the entire project was generated directly by Codex, with a focus on fully automated video editing.

The Core of RoughCut: A Fully Automated Editing Pipeline

RoughCut's positioning is very clear: handling video types with real footage, such as talking-head videos, unboxings, tutorials, screen recordings, and on-camera appearances. For "quick edit" types that rely on copyrighted material, the author admits this isn't his main focus, since professional platforms have a natural advantage when it comes to copyrighted footage.

The tool's core logic is full automation—users only need to select a target, and the system handles the remaining steps. Both visual style and language style support custom configuration. The system runs on Docker and starts automatically on boot, eliminating the need to repeatedly initialize the frontend and backend, making deployment and maintenance quite simple.

The significance of Docker deployment: Docker is a containerization technology that packages an application and all its dependencies into an isolated runtime environment, ensuring consistent behavior across different machines. Its core mechanism relies on the separation of "Image" and "Container": after developers package the runtime environment into an image, users simply pull the image and launch a container to get a runtime state identical to the development environment. For an individual developer's tool, this means users don't have to manually install Python dependencies, configure environment variables, or resolve version conflicts—a single docker compose up command starts the complete service. The "auto-start on boot" design (typically implemented via the restart: always policy) further lowers the barrier to entry, transforming the tool from "you need technical knowledge to run it" into an experience closer to consumer-grade software. This is why the author made Docker the core deployment approach for RoughCut, rather than the traditional Python script installation method.

In terms of editing logic, the author combined years of manual editing experience with video editors and built in a large number of editing rules and presets. For example: automatically correcting tilted footage from vertical smartphone shots via auto-rotation; intelligently identifying and removing clips that should be cut. Although a manual timeline alignment mode is retained, the author says "if the automatic results are good enough, this feature basically goes unused."

Three Editing Modes and Automated Tasks

RoughCut offers three main editing modes, plus an automated task system.

All-in-One Editing Mode

This is the author's most frequently used mode—import a video and start editing directly. This mode supports configuring personal account information, narration style, execution method, and more, with a rich set of feature options:

• Sound effect cues, automatic chapter division, progress bar and other basic packaging
• Script rewriting + TTS re-recording: suitable for users with limited spoken language organization skills. The system rewrites the script and re-records the voiceover using TTS (the author says the current results are somewhat stiff, so it's not used often)
• Bilingual versions: can automatically generate bilingual content such as Chinese-English or Chinese-Japanese
• Digital human narration: supports embedding a narration box; the author used a "Hun'er" digital human avatar

Explanation of TTS and ASR technology: TTS (Text-to-Speech) and ASR (Automatic Speech Recognition) are the two core technical pillars of video editing automation. ASR is responsible for transcribing human speech in videos into text, and is a prerequisite for "intelligently identifying clips to remove" and for the "commentary/derivative creation mode" to automatically match footage; mainstream ASR solutions currently include OpenAI Whisper (open-source, local deployment), Alibaba Cloud FunASR, and iFlytek speech, among others. Whisper, with its strong generalization ability for Chinese dialects and technical vocabulary, has become the top choice for localized automation tools. TTS re-synthesizes the rewritten script into speech, powering features like script rewriting voiceovers and bilingual version generation. In recent years, deep learning-based neural network TTS models (such as Microsoft Azure TTS, ElevenLabs, and CosyVoice) can clone a specific speaker's timbre, pace, and emotional style from just a small amount of reference audio. This is precisely the technical foundation for the "reference voice" configuration item in creator management—the system learns the creator's own vocal characteristics, making the synthesized speech more personally recognizable and avoiding a mechanical feel.

Background on digital human technology: Digital human (Digital Human / Virtual Avatar) technology uses deep learning "audio-driven facial animation" models to precisely lip-sync a pre-recorded real-person appearance with real-time generated speech, producing realistic virtual presenter videos. Its core technical approaches fall into two categories: one is based on 2D image-driven solutions (such as SadTalker and MuseTalk), which have lower computational costs and are suitable for local deployment; the other is based on 3D modeling for full-body digital humans (such as HeyGen, D-ID, and Silicon Intelligence), which produce more realistic results but usually require cloud API calls. In content creation scenarios, digital humans can replace on-camera appearances, addressing situations where creators don't want to show their faces or lack the energy to film frequently, while maintaining visual consistency in content. The "Hun'er" digital human avatar in RoughCut is a custom virtual IP integrated into the editing pipeline via an embedded narration box—a typical example of incorporating a digital human toolchain into an automated workflow.

Smart Director Mode

Similar to the "one-click video generation" feature in other products. Out of personal preference, the author only maintained it to the level of "getting the pipeline to run through" without deep development. In the demo, he verified the pipeline's feasibility using the example "make a 2026 World Cup-themed video," but since no footage API was integrated, PowerPoint slides were used as a stand-in for the visuals. He frankly stated that he isn't a fan of this direct video-generation approach, believing that without high-quality footage, the results fall short of the AI tools in premium video editor subscriptions.

Commentary/Derivative Creation Mode

This is the most impressive mode, specifically designed for producing derivative content like "watch a movie in three minutes" or "press conference quick recaps." After users import a script, the system automatically synthesizes narration via TTS, cuts and splits the original footage, and intelligently matches the corresponding clips during the narration. Even more noteworthy, the system will automatically pop up bubbles and special effects to emphasize key points, and automatically insert the corresponding footage from the original video when a particular topic is mentioned, producing a smooth and natural overall result.

The technical implementation of this mode relies on "semantic alignment" capabilities—the system needs to understand the semantics of the narration and locate the best-matching video clip in the original footage's ASR transcript, which is essentially a cross-modal semantic retrieval and timeline alignment problem. This explains why this mode places higher demands on AI reasoning than the others, and why the "naturalness" of the final result depends more heavily on the depth of understanding of the underlying language model.

Automated Tasks

Simply specify a folder and configure editing specifications, and the system can automatically batch-process all videos in the folder. It also supports smart merging—for example, a video on the same topic shot in three segments can be automatically merged and then edited as a whole, making it ideal for batch production of series content like unboxings.

Semi-Automated Publishing and Creator Management

RoughCut has a built-in publishing system that the author describes as "semi-automatic." After clicking "Generate Assets," the system automatically generates titles and descriptions, and calls Codex or Jimeng to generate a Chinese cover image (the cover image is automatically matched after the Agent recognizes the content of the original footage).

The publishing stage uses an "auto-generate + manual-publish" strategy. In the author's tests, publishing one round across all platforms takes no more than 10 minutes, with almost no thinking required. He specifically notes: Xiaohongshu video posts must be published manually; while platforms like Douyin, Bilibili, YouTube, and X support automatic publishing, if a platform has relevant restrictions, it's recommended to follow the rules and operate manually.

For creator management, the system supports configuring multiple creator profiles, each of which can independently set watermarks, digital human avatars, reference voices (for TTS), intro/outro assets, music libraries, and more. At the task strategy level, details such as editing schemes, script styles, and visual schemes (subtitle styles, cover styles) are specified.

There is also a knowledge base module that stores the methodology system accumulated by the author—these creation methodologies come from various projects and web collection, and the system uses this content to continuously optimize its editing logic while also storing frequently used hotwords to improve ASR recognition accuracy.

Knowledge base and RAG technology: The "knowledge base" module mentioned in the article is typically implemented using the RAG (Retrieval-Augmented Generation) architecture. The core RAG process has two stages: in the offline stage, documents are split into semantic chunks and converted into high-dimensional vectors via an embedding model, then stored in a vector database (such as Chroma, Milvus, or Qdrant); in the online stage, when the AI makes a decision, the current task description is also vectorized for similarity retrieval, and the most relevant knowledge chunks are inserted into the prompt context. This mechanism turns the creator's accumulated methodology documents, hotword lists, and other content into "external memory" that the AI can retrieve, making model output better aligned with a specific creator's style preferences and domain knowledge—without fine-tuning the model itself. Compared with hard-coding all knowledge into a fixed prompt, the RAG approach is more flexible and scalable, and is the mainstream technical path for AI workflow systems to achieve "personalization" and "continuous learning" today. The hotword library acts directly on the ASR recognition layer, reducing transcription error rates by boosting the decoding weight of proper nouns (such as product names, personal names, and industry terms), thereby improving the accuracy of all subsequent text-dependent automation steps.

The Bigger Picture: A Series of Productivity Projects

The author reveals that RoughCut is just one component of his overall plan. The main project "Hefeili" is positioned as image editing, RoughCut handles video editing, and there are also educational projects, plus "Hydra Metric," which he has spent half a year developing as a key focus. These projects will be open-sourced one after another, including a small "virtual friend" project.

The author also candidly admits that RoughCut has not yet reached the ideal state of "one-click configuration for beginners"—although it can be deployed directly via Docker, many automation configuration steps still need refinement. Configuring models and routing currently requires users to have some understanding of Agents, and he plans to turn this into a smarter automatic configuration solution in the future. He also plans to continue improving the screen-recording editing experience, with detailed designs such as mouse trail zoom.

Conclusion

RoughCut represents the pragmatic transition of AI Agent tools from "demo toys" to "productivity tools." Its value lies not in flashy technical showmanship, but in directly tackling creators' most painful editing barrier—dramatically reducing the time cost of publishing videos through a fully automated editing pipeline and a semi-automated publishing system. Under the hood, it integrates multiple mature AI infrastructure components including Docker containerized deployment, ASR speech recognition, TTS speech synthesis, digital human technology, and RAG knowledge base retrieval, and connects these capabilities into an automated workflow that is transparent to the creator through Agent orchestration. Although there is still room for improvement in usability and one-click deployment, its product philosophy of "starting from real needs and integrating open-source capabilities" offers considerable reference value for creators looking to leverage AI tools to improve content production efficiency.

Key Takeaways