Vibecoding in Practice: Task Concurrency Splitting Makes AI Inference 3x Faster

From 100 Seconds to 30 Seconds: A Real-World Concurrency Refactor

In AI application development, performance is often the critical bottleneck that determines user experience. Bilibili creator "Powang" shared a typical performance optimization case in episode 134 of his [AI Practical] series: by splitting tasks and designing concurrent execution, he compressed the processing time for a script performance guidance tool from around 100 seconds to around 30 seconds — achieving roughly a 3x speedup.

The value of this case lies not just in the result, but in the clear, reusable "task concurrency" methodology it demonstrates. This approach is applicable to any scenario that requires calling a large model for deep reasoning.

Where It Starts: Deep Reasoning Is Too Slow

In the previous episode, the author had already integrated performance guidance prompts for Alibaba Cloud's engine and Volcano Engine, splitting them into calls to two different models. But a new problem emerged: performance guidance requires a fairly deep reasoning process, making the overall latency quite significant.

Take a script of around 1,000 characters as an example — the reasoning process can take 60 to 120 seconds, roughly "1 second per 10 characters." This kind of latency is clearly unacceptable for an interactive tool. Since having a single Agent analyze everything from start to finish is too burdensome, reducing single-point pressure and introducing concurrency becomes a natural optimization direction.

Why are reasoning models so slow? Inference latency in large language models is determined by several factors: model size (number of parameters), number of output tokens, and whether Chain-of-Thought (CoT) reasoning is enabled. For tasks that require "deep reasoning," models often need to generate a large number of intermediate reasoning steps internally, which themselves consume token quota and compute time. In reasoning-enhanced models like DeepSeek-R1 and the o1 series, "thinking tokens" can far exceed the final output, causing single-call latency to be much higher than ordinary completion tasks. The empirical observation of "1 second per 10 characters" actually reflects the typical throughput of these models under high load — in shared API environments, latency increases further when many concurrent users are present.

Core Idea: Split Tasks by "Act" and "Character"

The core of this episode is having AI redesign a concurrency scheme. The original approach was "analyze all decisions in one shot"; it was now changed to fine-grained splitting based on script structure.

Why Can It Be Split? Two Natural Dimensions

The author identified two reasonable splitting dimensions — and these are the key to why the whole scheme works:

First, split by "Act." A script is typically divided into multiple acts, with low coupling between acts — each act has its own complete logical structure. This natural boundary makes parallel processing possible: analysis of different acts doesn't interfere with each other.

Second, split by "Character." This is especially clever. Different characters correspond to different voice timbres, different timbres correspond to different engines (Alibaba Cloud / Volcano Engine), and different engines determine different performance guidance styles. Therefore, each character is naturally an independent unit for a large model call.

The engineering difference between concurrency and parallelism Concurrency and Parallelism have a subtle distinction in engineering: concurrency means multiple tasks alternate in progress over time, while parallelism means multiple tasks physically execute simultaneously. When calling external APIs, the bottleneck is network I/O rather than local CPU — even single-threaded async concurrency (e.g., Python's asyncio) can significantly improve throughput, since the CPU can initiate other requests while waiting for an API response instead of blocking. This is fundamentally different from CPU-intensive tasks that require true multi-core parallelism. Therefore, "concurrency" optimization in AI applications is typically closer to I/O concurrency rather than compute parallelism — it's low-cost to implement but delivers substantial gains.

From 1 Task to 14 Concurrent Streams

Combining both dimensions, the split strategy is "Act × Character." For example: if a script has three acts — 5 characters in Act 1, 4 in Act 2, and 5 in Act 3 — you can decompose it into up to 5+4+5=14 parallel tasks.

Note that this isn't simply splitting three acts into three tasks. Each character within each act is treated as an independent processing unit. This fine-grained decomposition is the core of the speedup.

Engineering Constraints on Concurrency: Hardware Is the Ceiling

So can you just naively spin up 14 concurrent streams? The author says no — more concurrency isn't always better; it should be determined based on hardware capacity.

A Pragmatic Rule of Thumb

The author mentions that the current design follows something like CPU core count × 2 — for example, an 80-core machine supports up to 16 concurrent streams. He honestly admits: "I don't know exactly how that number is derived, but I can ask AI to find the best practice."

Where does the concurrency limit come from? The rule of thumb "CPU cores × 2" comes from traditional OS thread scheduling: for I/O-intensive tasks, threads actively yield the CPU while waiting for I/O, so each core can serve multiple threads simultaneously without contention. Common Python constructs like ThreadPoolExecutor and asyncio Semaphore follow similar upper-limit logic. However, when calling large model APIs, the real bottleneck is usually not local hardware but the API provider's rate limits — typically measured in RPM (requests per minute) or TPM (tokens per minute). Therefore, for AI applications, the concurrency limit should reference the API documentation's rate limits rather than local machine specs.

This statement also illustrates a typical Vibe Coding working style: developers don't need to master every underlying detail — instead, they ask AI to find best practices and implement them, then address issues as they arise.

What is Vibe Coding? Vibe Coding is a development paradigm proposed by Andrej Karpathy in 2025 that quickly gained popularity. Its core characteristic is that developers describe their intent in natural language, AI handles the concrete implementation, and developers focus on direction and problem diagnosis rather than low-level details. This mode is extremely efficient for prototyping and personal projects, but in production systems requiring high reliability and strong consistency, professional engineers still need to rigorously review AI-generated output — because AI's "best practices" may not suit the specific runtime environment and business constraints.

Toy-Level vs. Enterprise-Level Trade-offs

The author emphasizes an important mindset: this is a "toy-level" personal project, not an enterprise-level system. Since only one machine is running it with no resource contention, precise tuning of the concurrency count "doesn't matter at all" — just go ahead and try it.

But he also warns that if you're building an enterprise-level system, this rough approach won't hold up — you'd need professional engineers with a deeper understanding of hardware. This clear awareness of scenario boundaries reflects real-world engineering experience: don't incur enterprise-level engineering costs for a toy-level project.

Full Workflow: Schedule → Concurrent Execute → Review

Although 14 parallel tasks are split out, the overall workflow actually involves 16 calls. Behind this number lies a complete "schedule—execute—reassemble" loop:

1. Pre-scheduling (1 call): First do an overall scheduling pass to uniformly process characters, acts, dialogue, and other information as a prerequisite for concurrency.
2. Parallel execution (14 streams): Launch large-scale concurrent calls by act and character. Each task returns at different times but none block the others.
3. Automated review (1 call): Results returned by different Agents are reassembled back into the unified full script. During reassembly, a review is performed, and any rough spots are addressed with technical tuning.

This "distribute—parallelize—aggregate" pattern is essentially a classic MapReduce approach applied to AI applications.

The connection between MapReduce and multi-Agent collaboration MapReduce is a large-scale data processing paradigm proposed by Google in 2004. Its core idea is to break large tasks into parallelizable Map sub-tasks, then aggregate results via a Reduce step. This idea has had a profound influence on the distributed computing field — frameworks like Hadoop and Spark are built on this foundation. In this case, "pre-scheduling" corresponds to the data preprocessing stage, "14 concurrent streams" corresponds to the Map phase (each act × character combination is an independent map operation), and "review and reassembly" corresponds to the Reduce phase (merging fragmented results into a unified script). In recent years, as LLM applications have taken off, similar patterns have been repackaged as "multi-Agent collaboration" frameworks — but the underlying logic traces directly back to MapReduce. Understanding this historical lineage helps developers draw inspiration from the wealth of distributed systems design experience.

The key insight: since the script has an overall global structure, the locally parallelized results can be reassembled back into a global whole, with the review step ensuring consistency.

Results and Reusable Methodology

The optimization results are significant. The original processing took about 100 seconds for 1,000 characters of dialogue; after the refactor, whether the script is 1,000, 2,000, or 3,000 characters, it completes in roughly 30 seconds.

The time breakdown becomes clear: pre-scheduling takes about ten-plus seconds, concurrent execution takes another ten-plus seconds, and the final review takes some additional time — totaling around 30 seconds. More importantly, because the middle section runs in parallel, total latency doesn't grow linearly as the word count increases — this is the core dividend that concurrency delivers.

A Reusable AI Performance Optimization Methodology

Setting aside the specific script scenario, this case offers several general takeaways:

• Find natural boundaries in the task: Dimensions with low coupling and independent processability — like "acts" and "characters" — are the best entry points for concurrency splitting.
• Reduce single-Agent burden: Rather than having one Agent carry a heavy end-to-end task, break it into multiple focused small tasks and process them in parallel.
• Don't forget aggregation and review: Concurrent execution produces fragmented results; there must be a reassembly and quality-check step to ensure overall quality.
• Know your limits: Concurrency should be constrained by hardware capacity (and API rate limits); toy-level projects don't need over-engineering.

For developers building personal tools or prototypes with AI, the "split tasks → fire concurrently → then aggregate" approach is a practical path to squeezing more performance out of large models and improving user experience.

Key Takeaways