Open-Source Video-to-Text Tool: Dual-Mode Approach with Subtitle Extraction + Gemini AI Transcription

Project Overview

In everyday learning and work, converting video content to text is a high-frequency need — whether it's organizing meeting notes, taking course notes, or repurposing content. But videos vary widely: some come with subtitles, while others have no text information at all.

The Video Transcription market has grown rapidly in recent years. According to industry reports, the global speech recognition market is expected to exceed $50 billion by 2030. Multiple forces are driving this growth: the explosion of short-form video and online education has caused video content volume to surge dramatically; the normalization of remote work has created massive demand for organizing meeting recordings; and content accessibility regulations (such as the ADA in the US and EAA in the EU) require video content to provide text alternatives. On the technology supply side, from early traditional speech recognition based on HMMs (Hidden Markov Models), to end-to-end deep learning models (such as DeepSpeech and Whisper), to today's multimodal large models with direct audio understanding, transcription technology continues to improve in both accuracy and usability.

The GitHub open-source project claude-skill-video-transcribe addresses this pain point with a practical solution. It employs a strategy of "grab subtitles when available, use AI transcription when not," supports YouTube, Bilibili, and local video files, and uses Gemini 2.5 Flash as its core transcription engine to achieve efficient video-to-text conversion.

Core Design: A Dual-Path Strategy of Subtitle Extraction First, AI Transcription Second

Prioritizing Existing Subtitle Extraction

For videos on platforms like YouTube and Bilibili, much content already comes with manually created or auto-generated subtitles. Directly extracting these subtitles is not only faster but typically more accurate. The tool prioritizes attempting to fetch existing subtitle data from video platforms, eliminating unnecessary AI processing overhead.

From a technical implementation perspective, subtitle extraction from video platforms relies on APIs or page data interfaces provided by the platforms. YouTube's subtitle system is divided into two categories: manually uploaded human subtitles by creators, and machine subtitles generated by the platform through ASR (Automatic Speech Recognition). Tools typically leverage open-source downloaders like yt-dlp to obtain subtitle track data, which is stored in formats like SRT and VTT containing timestamps and corresponding text. Bilibili's subtitle system is similar, supporting CC subtitles (community-contributed subtitles) and AI subtitles, with JSON-formatted subtitle data obtainable by parsing its Web API. The advantage of directly extracting platform subtitles is clear: human subtitles have been proofread and are far more accurate than real-time speech recognition; even auto-generated platform subtitles have typically been optimized by platform-level models, ensuring a baseline quality.

Gemini 2.5 Flash AI Transcription as Fallback

When a video has no available subtitles, the tool automatically switches to AI transcription mode, calling the Gemini 2.5 Flash model to transcribe the audio content.

Gemini 2.5 Flash is a multimodal large language model launched by Google DeepMind in 2025, a version within the Gemini 2.5 series optimized for high-throughput, low-latency scenarios. Compared to Gemini 2.5 Pro in the same series, the Flash version significantly reduces inference cost and response time while maintaining high reasoning quality. As a natively multimodal model, it can directly process audio input rather than using a traditional pipeline approach — which typically requires first converting speech to text via an ASR model, then having a language model understand and organize it. Gemini 2.5 Flash's end-to-end processing approach means the model can simultaneously leverage acoustic features and semantic context, offering clear advantages when handling complex scenarios like colloquial expressions, specialized terminology, and multilingual mixing. Its API is available through Google AI Studio or Vertex AI platforms, billed per token, with Flash version pricing at a fraction of the Pro version.

This dual-path strategy is highly pragmatic: use existing subtitles when available to avoid wasting compute, and have a reliable backup when AI intervention is needed.

Supported Video Sources

The tool covers the three most common video-to-text scenarios:

• YouTube: The world's largest video platform, with massive English and multilingual content
• Bilibili: A mainstream Chinese video platform, rich in Chinese-language content
• Local files: Direct processing of locally stored videos, suitable for private content like meeting recordings and course captures

Whether you're studying overseas technical videos, organizing domestic course content, or processing locally recorded material, it's essentially all covered.

Tech Stack and Implementation Details

The project is developed in Python, the most mainstream language in the current AI toolchain. Judging from the "claude-skill" prefix in the project name, the tool is likely designed as a skill plugin for Claude (Anthropic's AI assistant), capable of integrating with Claude's workflow.

Claude's skill plugin mechanism allows developers to encapsulate external tools as callable capability modules for Claude, enabling Claude to execute specific tasks within conversations — such as file processing, data analysis, and API calls. This design philosophy is similar to ChatGPT's Plugins and GPT Actions mechanisms, with the core idea being to let large language models serve as a "dispatch hub" that calls specialized tools to complete tasks they cannot handle directly. Under this architecture, users can directly request video transcription within a conversation with Claude, and Claude will invoke the skill plugin to complete subtitle extraction or AI transcription and return the results to the user. This human-AI collaboration model is becoming the mainstream paradigm for AI tool development — the so-called Agentic Workflow.

Choosing Gemini 2.5 Flash over traditional open-source speech recognition solutions like Whisper is a noteworthy technical decision. Whisper, open-sourced by OpenAI in 2022, is one of the most widely used speech recognition models today. It's based on the Transformer encoder-decoder architecture, trained on 680,000 hours of multilingual labeled data, and supports speech-to-text in nearly 100 languages. Whisper's advantages include being fully open-source, locally deployable, network-independent, and having a rich community ecosystem (such as faster-whisper and whisper.cpp acceleration implementations). However, Whisper is essentially a pure speech recognition model — its job is to map acoustic signals to text sequences, lacking deep contextual semantic understanding.

By comparison, multimodal large models like Gemini 2.5 Flash can leverage broader world knowledge for disambiguation during transcription — for example, distinguishing between identically pronounced but differently meaning technical terms, or more accurately switching language recognition in multilingual conversations. However, the trade-off is dependence on cloud APIs, incurring usage costs, and being subject to network latency and API quota limitations. A Google API key is required for configuration.

Use Cases

• Content creators: Quickly convert videos to text transcripts for secondary creation on blogs, social media, etc.
• Learners: Organize online course notes for review and keyword searching
• Researchers: Transcribe interviews, lectures, and other audio/video materials
• Developers: Use as part of an automated workflow for batch video-to-text processing

Summary

claude-skill-video-transcribe currently has a modest star count (31 Stars), but its design philosophy is clear and its functional positioning is precise. The dual-path strategy of "grab subtitles when available, use AI transcription when not" balances efficiency and coverage, and the choice of Gemini 2.5 Flash as the engine reflects cost-effectiveness considerations. If you need video-to-text capabilities — whether for YouTube, Bilibili, or local videos — this open-source tool is worth trying.