[KongchangAI]
· 1 min read· 959 words

JioStar Chief Architect: AI Discovery at 500 Million Users

JioStar Chief Architect: AI Discovery at 500 Million Users

JioStar's architect shares evals, guardrails, and production lessons for AI discovery at 500M users.

JioStar Chief Architect Vijay Seshadri shares how his team deployed AI discovery at 500 million users. Because scale amplifies every design flaw, the team treated evaluability, constraint, and rollback as core system properties from day one. Evals combine automated regression with human calibration across relevance, diversity, and freshness; guardrails layer input and output constraints tuned to business context; and production depends on progressive rollout, observability, and fast rollback. The takeaway: at scale, competitiveness comes from the evaluation and constraint systems around the model, not the model itself.

From Lab to 500 Million Users: The Scale Challenge of AI Discovery

When an AI product serves 500 million users instead of a lab test set, the center of gravity in engineering decisions shifts fundamentally. JioStar Chief Architect Vijay Seshadri recently shared his team's experience deploying AI discovery features at hyperscale, focusing on evaluation systems, guardrails, and the path to production.

JioStar Chief Architect on AI Discovery at 500 Million Users

AI discovery refers to using model capabilities to help users find what they actually want to watch amid vast content catalogs. At JioStar's user scale, the hard part isn't whether the model itself is smart enough—it's that every recommendation and every generation must complete in a very short time window, without producing errors that embarrass users or the platform.

Seshadri's core judgment: scale amplifies every design flaw. A strategy that performs well at low traffic can rapidly fail in front of 500 million users due to long-tail distributions, edge cases, and adversarial inputs. So from the start, the team treated "evaluable, constrainable, rollback-ready" as fundamental properties of the AI discovery system—not engineering details to patch in after launch.

Evaluation Systems: Turning "Is It Good?" into a Measurable Question

Seshadri emphasizes that evals are the first gate AI discovery must pass to move from demo to production. Unlike traditional recommender systems that rely on offline metrics, generative AI discovery experiences are hard to summarize with a single number. Teams need to track relevance, diversity, freshness, and user satisfaction simultaneously, and build evaluation pipelines that can run continuously.

A key practice he cites is combining human review with automated evaluation: automation handles high-frequency regression, while human review calibrates whether the evaluation itself has drifted. This combination matters especially at hyperscale, because pure human review can't cover the traffic volume, and pure automation tends to devolve into metric gaming.

Evals in the generative AI context refer to a methodology for judging model output quality using a set of repeatable test cases. Unlike traditional classification models that can be measured with single metrics like accuracy or AUC, generative tasks produce open-ended text or multimodal content that can't be compared against fixed labels. Evaluation therefore typically combines "LLM-as-a-judge" (using another model to score against rubrics), rule-based checks, and human annotation. Eval sets must cover both typical scenarios and long-tail edge cases, and be continuously updated as online traffic distributions shift. In hyperscale products, the compute cost and latency of eval pipelines are themselves engineering constraints—teams often need sampled evaluation running in parallel with full-volume monitoring to control costs while catching regressions in time.

Guardrails: Blocking Risk Before Users See It

Guardrails were another focal point in Seshadri's talk. At 500 million users, any inappropriate content output can be amplified rapidly. JioStar's approach is to layer constraints throughout the AI discovery pipeline rather than relying solely on the model's own alignment capabilities.

These constraints cover both input and output: filtering obviously anomalous or malicious requests on the input side, and running safety and compliance checks on generated results on the output side. Seshadri specifically notes that guardrail design must match the business context—too loose and risk spills over, too strict and you kill the diversity and serendipity that make discovery valuable. Finding that balance is itself a product decision.

Guardrails are a runtime control layer independent of the main model, used to intercept or correct unsafe inputs and outputs. Common implementations include: input-side prompt injection detection and sensitive-word/malicious-intent classification; output-side content safety classifiers, factual consistency checks, and format/compliance rule checks. Compared with relying on the model's own alignment (such as refusal tendencies trained via RLHF), external guardrails offer the advantages of independent updatability, auditability, customization for specific business policies, and resilience across model upgrades. The cost is added inference latency and engineering complexity, so guardrails are typically deployed in tiers—high-risk scenarios get strict multi-stage checks while low-risk scenarios get lightweight filtering only, balancing safety and experience.

Toward Production: Engineering Lessons from AI Discovery

Bringing AI discovery into production, Seshadri's experience boils down to a few pragmatic engineering principles. First, progressive rollout: validate evals and guardrails in controlled traffic before gradually expanding coverage. Second, maintain observability so teams can see model behavior changes in real time. Third, prepare rollback paths—any AI feature should be able to exit quickly when something goes wrong.

None of these practices are novel, but at 500 million users, each one gets stress-tested to its limits. The value of Seshadri's talk is that he grounds these principles in real business constraints: evals must be runnable, guardrails must actually hold, and production systems must withstand the randomness that scale introduces.

Progressive Rollout means gradually releasing a new feature by user percentage or traffic bucket rather than launching to everyone at once. A typical approach expands from internal employees to 1% of traffic, then 5%, with each stage gated on core metrics and guardrail trigger rates. It's paired with Observability: logging, metrics, and tracing that record model input/output distributions, latency, failure rates, and guardrail interceptions, so teams can catch anomalies before problems spread. Rollback requires features to be toggleable, so that degraded metrics can switch back to the old strategy in seconds. Together, these three form the standard engineering paradigm for safely launching AI features in hyperscale systems.

Implications for AI Product Teams

JioStar's case shows that the competitiveness of large-scale AI products often depends not on model parameters but on the evaluation and constraint systems built around the model. For teams pushing AI features to more users, Seshadri's experience offers a reference framework: build solid evals first, embed guardrails into the pipeline, then use progressive rollout and observability to keep risk within acceptable bounds. Scale isn't a barrier to AI deployment—it's the touchstone that tests engineering maturity.

#AI discovery#JioStar#evals#guardrails#progressive rollout#LLM-as-a-judge#hyperscale AI
Share:

Related articles