An Open-Source Pipeline Turns Lectures into Text, Summaries, Audio and Braille at Once

Lectures are one of the most common ways to learn in a classroom, yet also among the least accessible. Students with hearing impairments can't follow the audio, students with visual impairments can't read the slides, and students with cognitive or attention difficulties often find a 60-minute recording impossible to approach. Existing accessibility tools tend to operate in silos—one product for transcription, another for summarization, and yet another process for braille. Developer Vishal Mundewadi tried to bridge this gap: he built AccessLearnAI, where you upload a lecture audio file and get back a text transcript, a concise summary, synthesized speech audio, and braille output all at once.

The value of this story isn't just "another accessibility tool"—it's that it demonstrates an end-to-end engineering path: stringing four open-source models into a pipeline, and hitting the real-world wall of deployment along the way.

One Pipeline, Four Output Formats

AccessLearnAI's design philosophy is straightforward: rather than building a separate feature for each disability, let a single input generate all formats. The pipeline consists of four relay stages—Whisper handles speech-to-text, T5 handles summary compression, edge-tts converts the summary back into natural speech, and the final step converts text into braille.

The output of each stage is the input to the next. The developer chose Whisper because it's open-source and robust across accents and recording quality—especially critical for real classroom recordings. T5 compresses the verbatim transcript into a summary readable in a few minutes. edge-tts provides natural-sounding speech. The braille generation targets blind and low-vision users. This "single input, multiple outputs" structure is closer to real needs than splitting features by disability type—the same student might need both a summary and audio.

Whisper is an automatic speech recognition (ASR) model open-sourced by OpenAI in 2022, using an encoder-decoder Transformer architecture trained on 680,000 hours of multilingual, multitask weakly supervised data. It comes in multiple sizes from tiny to large, with parameter counts ranging from about 39 million to 1.55 billion, allowing developers to trade off accuracy against resource budget. Whisper's standout feature is its strong robustness to accents, background noise, and recording quality variations, plus support for language identification and translation, making it commonly used directly in real-world non-ideal recording scenarios like classrooms and meetings. Its limitations include high GPU memory usage for large models, the need to segment long audio, and inference latency noticeably higher than lightweight ASR solutions.

T5 (Text-to-Text Transfer Transformer) is a unified text-to-text framework proposed by Google that formulates summarization, translation, classification, and other tasks as "input text → output text." The largest original T5 version has about 11 billion parameters, while practical deployments typically use smaller variants like T5-small, T5-base, or Flan-T5 to control memory. Summarization usually employs abstractive generation—reorganizing language rather than extracting original sentences—so it must work in concert with transcription quality; typos and spoken redundancy in transcripts directly affect summary coherence.

Three-Layer Architecture: FastAPI, React, and Containerized Deployment

The system is architecturally divided into three layers: the backend uses FastAPI with PostgreSQL for storage, the frontend is React with TypeScript, and deployment runs Docker containers on GCP with the frontend hosted on Vercel.

FastAPI was chosen for a practical reason—the ML code itself is written in Python, so FastAPI can expose each pipeline stage directly as clean API endpoints without adding another conversion layer between models and web services. For a project centered on chaining models together, reducing glue layers means fewer failure points and shorter debugging paths.

The Really Hard Part: Running Four Models on a Free Tier

The pipeline ran smoothly on the development machine, but things were different after going live.

After deploying the backend to Render's free tier, out-of-memory errors occurred repeatedly. The reason wasn't complicated: Whisper and T5 were loaded into memory simultaneously, and together they exceeded the free tier's limit. This issue never surfaces during local development—the dev machine has ample memory, so sequential loading or parallel residency goes unnoticed; once in a resource-constrained production environment, model memory occupancy becomes a hard constraint.

This pitfall is worth noting for anyone deploying ML applications: a model that runs locally doesn't mean it can run in production. Deploying an ML system requires managing memory as much as managing code. Viable directions include on-demand model loading, releasing memory between stages, splitting heavy models into separate services, or simply abandoning the free tier for an option with guaranteed memory—but the original article doesn't elaborate on these trade-offs, leaving them as engineering decisions for readers to judge for themselves.

OOM (Out of Memory) is one of the most common runtime failures when deploying ML services. In containerized or PaaS free-tier environments, the memory available to a process is typically hard-limited (e.g., 512MB or 1GB). Once the combined usage of model weights, inference intermediate tensors, and framework runtime exceeds the limit, the process is terminated by the OS or container runtime. Even at smaller sizes, Whisper and T5 often each occupy hundreds of MB to several GB of memory after loading, and having both resident simultaneously easily hits the ceiling. Common mitigation approaches include: using quantized or distilled smaller models, lazy-loading on request with explicit memory release between stages, splitting different models into separate services to isolate memory peaks, and choosing paid instances with memory guarantees. These approaches involve trade-offs in throughput, latency, and cost, and need to be evaluated against specific traffic patterns.

Open Source and Online Demo

The project code is open-sourced on GitHub (vishal-mundewadi/AccessLearnAI), with an online demo available on Vercel. For developers wanting to replicate a similar pipeline, this is a cleanly structured, manageable-scale reference implementation: four stages, three-layer architecture, and a documented deployment pitfall.

More worth pondering is the question it raises—accessibility shouldn't be four disconnected products, but can be four parallel outputs on a single pipeline. This approach applies equally to edtech, meeting transcription, public content distribution, and other scenarios.

Frequently Asked Questions

What input formats does AccessLearnAI support? The original material only mentions "uploading lectures" and doesn't specifically list supported audio formats; refer to the open-source repository documentation for details.

Must all four models be loaded simultaneously? The original failure experience precisely demonstrates that simultaneous loading exceeds free-tier memory; actual deployment requires on-demand loading or service splitting, though the author doesn't provide specific remediation plans.

Is the braille output physical braille or text encoding? The original only states "converted to braille for blind and low-vision users" without specifying the concrete output format.

One-line Summary

AccessLearnAI uses a four-model pipeline to turn lecture audio into transcripts, summaries, speech, and braille.

Paragraph Summary

The article introduces AccessLearnAI, an accessibility learning tool built by developer Vishal Mundewadi. It adopts a "single input, multiple outputs" approach, feeding lecture audio sequentially through four stages—Whisper speech-to-text, T5 summary compression, edge-tts speech synthesis, and braille conversion—producing transcripts, summaries, audio, and braille simultaneously, avoiding separate products for different disabilities. The system comprises a FastAPI+PostgreSQL backend, a React+TypeScript frontend, and a deployment layer of Docker on GCP plus Vercel. The article highlights a real engineering problem encountered during deployment: loading Whisper and T5 simultaneously exceeded Render's free-tier memory, causing OOM, demonstrating that locally runnable models don't guarantee production viability. The project is open-sourced with an online demo and serves as a reference implementation for multi-model chaining applications.