[KongchangAI]
· 3 min read· 1,529 words

How Mercor Powers Thousands of Internal Apps and 63 Billion Tokens on One Platform

How Mercor Powers Thousands of Internal Apps and 63 Billion Tokens on One Platform

Mercor uses a unified gateway and identity layer to solve AI scaling's capacity, cost, and permission bottlenecks.

Mercor provides expert human data for frontier AI labs under bursty demand and 24-hour delivery windows. Vercel AI Gateway gave it more inference capacity and cross-vendor failover, cutting batch tasks from over 12 hours to 2–3 hours and routing 18M requests and 63B tokens in 10 days without rate limits. Engineers hold keys with expiration, token limits, and spending caps, with usage traceable by team and project. Business teams' projects grew from a dozen to ~2,000; Passport provides unified auth, and Audiences uses IdP groups for second-level provisioning and automated offboarding. Agents use least privilege, with model access and system operations governed separately.

Mercor platform architecture overview

AI data company Mercor connects hundreds of thousands of domain experts with frontier AI labs, supplying high-quality human data for model training and evaluation. When business demand arrives in bursts and each evaluation task has only a 24-hour delivery window, model capacity and access management become the critical bottlenecks determining customer experience. Mercor's solution was to consolidate both model calls and identity authentication onto a single platform, turning the security team from approvers into enablers.

From 12 Hours to 2 Hours: Breaking the Capacity Bottleneck

Mercor previously used a mix of direct integrations and third-party gateways, which couldn't provide enough tokens-per-minute capacity for open-source model workloads during demand peaks. Requests hit vendor rate limits and entered retry queues, stretching time-sensitive batch tasks to over 12 hours—consuming more than half of customers' 24-hour deadlines.

Vercel AI Gateway changed two things: more inference capacity, and multi-vendor failover routing when requests hit rate limits. Batch tasks no longer get stuck in retry queues but can switch between vendors and keep executing. According to Mercor, these tasks now complete in 2 to 3 hours. In the first 10 days after launch, the team routed 18 million requests and 63 billion tokens through the Gateway without hitting rate limits.

Mercor didn't put all its eggs in one basket—it still uses multiple gateways and vendors, valuing Vercel's packaging of model access, control capabilities, and hands-on support. For engineers, a single API connects to major foundation model vendors and a growing catalog of open-source models, so trying new models no longer requires writing a separate set of vendor adapter code.

AI Gateway is an intermediary layer between applications and model vendors that unifies differences across vendor APIs and provides routing, rate limiting, retries, and observability. Its core value is abstracting "which vendor to use" from code into a runtime decision: when a vendor returns 429 (rate limit) or 5xx, the gateway can automatically switch to a backup vendor and continue, rather than letting requests block in retry queues. For evaluation tasks with 24-hour delivery windows, this multi-vendor failover directly determines whether tasks complete on time. Additionally, gateways typically provide unified billing and usage records, making token attribution by team and by project possible.

Where Tokens Go: Giving Experiments Budgets and Deadlines

The more models are used, the harder bills are to explain. Mercor's previous setup offered almost no visibility into usage broken down by team and project, and a single loop error in a script could burn through the budget.

Now hundreds of engineers use open-source models through AI Gateway, each holding an independent key with an expiration time, token limit, and spending cap. The security team can grant an experiment a budget and a deadline, then use usage records to trace which user, project, and workload type is consuming resources. Engineers can switch models to test results under the same gateway without managing a new set of vendor credentials each time. This model puts "govern first, then open up" into concrete mechanisms: limits don't hinder experimentation—they make it sustainable.

Token limits and spending caps are two different granularities of quota control. Token limits typically refer to the total tokens consumable in a single run or cumulatively, suited to per-task or per-experiment-batch settings; spending caps are hard thresholds priced in currency that reject requests when reached. Combined with expiration times, the two form a "temporary credential" model: the security team issues a key valid for a few days with a limited quota for an experiment, and the credential naturally expires when the experiment ends, with no manual revocation needed. Compared to long-lived shared vendor API keys, this model binds authorization scope, duration, and cost to the same credential—preventing script loop errors from burning through budgets while also enabling precise post-hoc tracing of consumption sources by key dimension.

Business Teams Build Their Own Apps: Projects Grow from a Dozen to ~2,000

Several hundred people on Mercor's business teams build apps on Vercel, and the company's project count grew from under a dozen to about 2,000. Many are temporary environments set up for specific customer collaborations.

A typical production scenario is letting experts view their own tasks and progress. Previously this required the engineering team to add each partner's interface into the core platform; now business teams build the interfaces themselves and share them with experts executing tasks.

The prerequisite for sharing is controlling who can log in—without building a separate authentication system for each project. Vercel Passport provides a unified authentication layer, handling login to Vercel deployments through existing identity providers. Mercor's identity team built a product called Audiences on top of it, integrating Passport as a first-class primitive to automate access provisioning: after selecting the experts who need access, Audiences automatically creates the IdP group, Vercel connection, and Passport configuration. According to Mercor, provisioning completes in seconds whether the audience is dozens of experts or tens of thousands. The offboarding flow runs in reverse with equal automation—since access is derived from the IdP group, deleting that group when a collaboration ends makes project access disappear with it, leaving no stale permissions behind.

IdP groups (identity provider groups) are a foundational primitive in identity governance: users are assigned to a group in an IdP (such as Okta or Azure AD), and downstream applications grant or revoke access based on group membership. Deriving access from IdP groups rather than maintaining separate allowlists on the application side has the advantage that permission lifecycle stays aligned with organizational identity lifecycle—when people join, change roles, or leave, you only adjust the group in the IdP and permissions across all downstream systems change in sync. This also explains why deleting an IdP group makes project access disappear: the application side holds no independent authorization records, so there's no stale residue of "person left but permission remains."

Drawing Clear Capability Boundaries for Each Agent

Beyond apps, Mercor also uses agents to answer questions from market experts, automate HR processes, and help employees navigate internal systems. Its approach is to configure each agent with narrow permissions matching its responsibilities rather than inheriting all of the caller's own permissions.

For agents using open-source models through AI Gateway, Mercor can also restrict the model range, set expiration times, and limit spending. Model usage and system operation permissions are governed separately, each with its own control point.

Least privilege means a subject is granted only the minimum set of permissions necessary to fulfill its responsibilities, rather than inheriting all of the caller's permissions. This is especially critical for agents: agents typically run as service identities, and if they directly inherit the triggering user's permissions, a prompt injection or behavioral deviation could expose the full scope of that user's capabilities. Mercor's approach is to define a permission boundary matching responsibilities for each agent individually and split "which models can be called" and "which internal systems can be operated" into two independent governance lines—the former restricts model range and spending through AI Gateway, the latter controls operation scope through system permissions, so tightening either dimension doesn't affect normal use of the other.

Make the Secure Path the Fastest Path

Mercor's next direction is to have more open-source models drive internal agents and systems, and to let more non-engineering teams build tools for experts. As scale grows, the company wants teams to spend less time configuring model integrations and app authentication so they can respond quickly to customer needs with tight deadlines.

As Mercor's head of security, Aaron Brown's thinking runs throughout: enabling more people to build means more simply giving them the access they need and continuously tracking how it's used. The goal is to make the secure path the fastest path—shared model access and repeatable authentication flows let experts get tools promptly and labs get results on time, while security workload doesn't have to grow at the same rate as the business.

Looking at Mercor's practice, the real barrier to scaling AI applications often isn't the model itself but the three pieces of "supporting engineering": capacity elasticity, cost visibility, and identity governance. Once these three are platformized, business teams dare to build freely, and the security team no longer has to repeatedly approve every new project.

One-line Summary

Mercor uses a unified gateway and identity layer to break through the capacity, cost, and permission governance bottlenecks of AI scaling.

Paragraph Summary

Mercor provides expert human data for frontier AI labs and faces dual pressure from bursty demand and 24-hour delivery windows. Through Vercel AI Gateway, the company gained more inference capacity and cross-vendor failover, compressing batch tasks from over 12 hours to 2–3 hours and routing 18 million requests and 63 billion tokens in the first 10 days without hitting rate limits. On cost, hundreds of engineers hold independent keys with expiration times, token limits, and spending caps, with usage traceable by team and project. On apps, business teams' self-built projects grew from a dozen to about 2,000; Passport provides a unified authentication layer, and Audiences relies on IdP groups to achieve second-level provisioning and automated offboarding. Agents use a least-privilege design, with model access and system operations governed separately—ultimately making the secure path the fastest path.

#Mercor#Vercel AI Gateway#Vercel Passport#identity governance#least privilege#token limits#AI scaling
Share:

Related articles