Open-Source Hub Adds Virtual Aggregated Provider: Cross-Provider Model Switching in Practice

Feature Overview: What Is a Virtual Aggregated Provider?

This update to the open-source Hub revolves around one core capability — the Virtual Aggregated Provider. In simple terms, it creates an abstraction layer on top of multiple real model providers, freeing users from being locked into a single API vendor. Instead, everything is accessible through a unified entry point, with model resources from different providers dispatched on demand.

For developers who juggle multiple API services, this aggregation layer solves a longstanding pain point: when you hold credits or quotas across several providers, how do you switch between them efficiently without constantly updating configuration in your code? The Virtual Aggregated Provider moves that switching logic into the platform itself.

Two Operating Modes: Specify a Model or Use Auto Dispatch

The new version offers two usage modes, each suited to different scenarios.

AOTL Auto Mode

Auto mode carries forward an implementation originally submitted by a community contributor via PR. In this mode, the system automatically dispatches requests across available providers based on its internal policy — no manual intervention required. This is ideal for users who want a hands-off experience: just configure your available providers, and let the system handle the rest.

Cross-Provider Aggregation for a Specified Model

Beyond Auto mode, the maintainer has now added cross-provider aggregation for a selected model. This means you can lock in a specific model (say, a particular version of a large language model) and have the Hub automatically cycle through all providers that support it. The scheduling granularity shifts from "pick any available provider" to "stay on this model, rotate across the providers that offer it."

The practical value here is real: the same model may be available from multiple providers, but each may differ in pricing, credit expiry status, and reliability. By specifying a model first and then aggregating across providers, users get both model consistency and provider-level flexibility.

Before diving deeper into these two modes, it helps to understand the technical context behind "Provider aggregation." Most mainstream LLM API providers — including OpenAI, Anthropic, Azure, and various domestic cloud vendors — expose OpenAI-compatible interfaces, which makes it possible to build a unified proxy layer on top of them. The Hub's Virtual Aggregated Provider is essentially a router: it exposes a unified API endpoint to upstream callers while managing authentication and dispatch policies for multiple real providers downstream. This architectural pattern is well-established in the open-source ecosystem — projects like LiteLLM and One API follow similar approaches — but this update refines the dispatch strategy to more closely match real-world user needs.

Prioritizing Near-Expiry Credits: Avoiding Quota Waste

The "sort by near-expiry credits" feature mentioned in the title is one of the most practical details in this aggregation logic. Many API providers assign expiration dates to credits or quotas — unused amounts simply expire. By sorting providers so that those with soon-to-expire credits are used first, Hub helps users consume their expiring balances before they go to waste.

This is a textbook example of designing from a real user pain point. For power users who have stocked up credits across multiple providers, manually tracking which account expires when — and using it in the right order — is nearly impossible. Letting the system automatically prioritize by expiry fills exactly that gap.

The "expiring credits" problem is particularly acute in China's AI API ecosystem, where many providers run marketing campaigns that bundle bonus credits with purchases — and those bonuses often come with 30-to-90-day validity windows. When using several providers in parallel, it's nearly impossible to maintain a mental expiry calendar. From a systems design perspective, sorting by expiry is essentially a weighted load-balancing strategy — where the weight is no longer just price or latency, but also incorporates a time-cost dimension. This mirrors the mindset used in operations for handling imminent certificate expirations or near-full disks: optimize for minimizing loss rather than maximizing performance.

Traffic Badge: Visualizing the Active Provider

Cross-provider switching introduces a new usability challenge: users can easily lose track of which provider is actually handling their current request. To address this, the maintainer added a real-time traffic badge — when the aggregated model switches to a particular provider, the interface displays that provider's badge so users always know where their traffic is going.

This visualization turns a black-box switching process transparent. Users can see at a glance which provider is serving the request, making it easier to troubleshoot issues, reconcile costs, and quickly identify the source of any anomalies.

The traffic badge addresses the observability challenge inherent in distributed call scenarios. Without such an indicator, users would have to review billing records or manually inspect network traffic to figure out where requests actually went — an inefficient process when diagnosing cost anomalies or service failures. Observability is a cornerstone of modern cloud-native systems, typically spanning logs, metrics, and traces. The traffic badge, while lightweight in form, directly addresses the single most frequent observability need at the UI level — attributing the current request to a source — and is especially user-friendly for those without a technical background.

Takeaway: Pragmatic Iteration in an Open-Source Project

This update reflects a healthy iteration pattern for open-source projects: Auto mode originated from a community PR, while the model-specific cross-provider aggregation and traffic badge were added incrementally by the maintainer. Together, they give this aggregation solution options for both the "set it and forget it" user and those who want fine-grained control.

For developers managing APIs across multiple providers, the Virtual Aggregated Provider — combined with near-expiry credit prioritization and the traffic badge — forms a reasonably complete solution for multi-source dispatch and observability. Since the project is open source, interested users can inspect the implementation details directly and even contribute.

Note: The above content is based on a feature demo published by a Bilibili content creator. For specific configuration instructions and the list of supported providers, please refer to the project's official documentation.