Whisper
Whisper is an automatic speech recognition (ASR) model developed and open-sourced by OpenAI that transcribes audio content into text and supports speech recognition and translation across multiple languages. Trained on large-scale multilingual speech data, the model exhibits strong noise robustness and can handle a variety of accents and speaking styles. It is applicable to use cases such as subtitle generation, voice assistants, and meeting transcription.
Core Facts
Whisper is an automatic speech recognition model open-sourced by OpenAI in 2022, built on a Transformer architecture and trained on 680,000 hours of multilingual audio data.
Azure OpenAI Service provides enterprise customers with the ability to invoke models such as GPT-4, DALL-E, and Whisper in a compliant environment.
The core design of the OpenAI API uses /v1/chat/completions as the main endpoint.
Timeline (last 90 days)
Modern speech transcription systems are mostly based on end-to-end deep learning architectures such as Whisper and Conformer, and their accuracy is already quite high in quiet environments, but in technical interview scenarios, specialized terminology (such as Kubernetes, Dijkstra), English abbreviations, accents and changes in speaking rate may cause recognition errors.
The larger the Whisper model, the higher the recognition accuracy, but the memory usage and inference latency also increase accordingly; for example, the large model may occupy more than 1 GB of storage, while the tiny model is only about 75 MB.
Running the Whisper model on iOS devices requires converting the model to Core ML or using inference frameworks such as whisper.cpp, and leveraging the Apple Neural Engine for acceleration.
For clear English speech, the WER of the Whisper large model can be as low as below 5%; for Chinese, due to word segmentation and homophone issues, WER is usually higher.
On a MacBook Air, transcribing one hour of audio takes only about 13 seconds, reaching about 277 times real-time speed.
On Whisper, int8 quantization usually has almost no effect on recognition accuracy, while int4 requires careful evaluation.
Achieving 13-second transcription of one hour of audio on a MacBook Air usually involves technical approaches such as model selection (the Whisper series or optimized versions), inference acceleration (Metal acceleration, Core ML conversion, etc.) and precision trade-offs (quantization such as int8, int4).
Punctuation and paragraphing are not native strengths of Whisper; the text it outputs often lacks sentence breaks, requiring post-processing models to restore punctuation and paragraph structure.
YTDLP and Whisper have become quite stable after years of iteration, Ollama has a very active community with frequent updates, while Hermes and Trailing PDF are relatively new, and the sustainability of their long-term maintenance remains to be seen.
YTDLP uses the Unlicense license, Whisper uses the MIT license, and Ollama uses the MIT license.
40 more timeline events
All Facts (20)
Whisper is an automatic speech recognition model open-sourced by OpenAI in 2022, built on a Transformer architecture and trained on 680,000 hours of multilingual audio data.
90%VerifiedAzure OpenAI Service provides enterprise customers with the ability to invoke models such as GPT-4, DALL-E, and Whisper in a compliant environment.
85%VerifiedThe core design of the OpenAI API uses /v1/chat/completions as the main endpoint.
80%VerifiedWhisper's architecture adopts an encoder-decoder structure, where the encoder transforms the mel spectrogram of audio into high-dimensional feature representations, and the decoder generates text output word by word in an autoregressive manner.
80%VerifiedWhisper is available in multiple sizes ranging from tiny (39M parameters) to large-v3 (1.5B parameters).
80%VerifiedThe core architecture of multimodal LLMs typically consists of three components: a visual encoder (such as CLIP or ViT), a cross-modal alignment layer, and a language decoder
80%VerifiedGroq has an advantage in large language model inference latency through its proprietary LPU (Language Processing Unit) chip.
70%VerifiedWhisperFlow is a free tool built on OpenAI's Whisper speech recognition model.
70%VerifiedApple provides the Speech Framework to support on-device speech recognition, but its capabilities are mainly limited to transcription and lack an LLM polishing step.
65%VerifiedThe Whisper model is available in multiple sizes ranging from tiny to large and can run on local devices.
65%VerifiedOpenAI's open-source Whisper model comes in multiple size variants, ranging from the tiny version with 39M parameters to the large-v3 version with 1,550M parameters, supporting nearly 100 languages.
65%VerifiedAlthough Whisper supports multilingual recognition, its performance on intra-sentential code-switching remains far below that in monolingual scenarios.
65%VerifiedModels such as Whisper and Wav2Vec 2.0 learn general acoustic representations on large amounts of unlabeled audio through self-supervised pre-training, reducing reliance on manually annotated data.
65%VerifiedOpenAI's open-source Whisper model achieves near-human-level word error rates across 99 languages and natively supports timestamp output.
65%VerifiedQuantized versions of Whisper (such as whisper.cpp and faster-whisper) can run speech recognition in real time on consumer-grade CPUs.
65%VerifiedText models supported by the Transformers framework include GPT, BERT, LLaMA, and T5; vision models include ViT, CLIP, and Stable Diffusion; audio models include Whisper and Wav2Vec; and multimodal models include LLaVA and Qwen-VL.
65%VerifiedWhisper基于Transformer编码器-解码器架构，在68万小时的多语言标注音频数据上进行弱监督训练
65%VerifiedWhisper, open-sourced by OpenAI, is a representative local ASR solution that can process audio in real time on consumer-grade GPUs.
65%VerifiedOpenAI released Whisper in 2022, based on a Transformer encoder-decoder architecture, with the encoder processing 80-dimensional Mel spectrograms.
70%UnverifiedOpenAI发布的Whisper模型支持包括越南语在内的近100种语言，且完全免费可本地部署
90%