How On-device AI Agents Solve the Context Problem of "Your Phone Should Already Know This"

When a user tells their phone's AI assistant, "Get me a ride to my usual airport," the natural expectation is: the phone already holds the calendar, contacts, location history, and the flight confirmation in email—so why do I have to repeat all of this?

The StareBrain team heard this complaint repeatedly during beta testing. Their answer: the problem isn't that the information doesn't exist, but that each app holds its own piece of data, and nothing connects them automatically. The core challenge of building an on-device AI Agent is establishing a context layer that runs locally—delivering the right information to the right action at the right moment, rather than uploading all data to the cloud for processing.

Two Types of Context: Action Context and Device Context

StareBrain splits the context an Agent needs into two categories. Action context describes what action the user requested, which parameters it needs, and which apps can fulfill it; device context describes the relevant information the phone already knows—upcoming calendar events, saved locations, recent search history, and so on.

Most Agent architectures already handle action context fairly well. The real complexity lies in device context: it's scattered across the OS and multiple apps, each with its own permission model. The team aggregates both into an AgentContext data structure, where DeviceContext contains upcoming events, saved locations, recently used apps, and summaries of active notifications.

The key design decision: the Agent must resolve parameters from device context before action dispatch—not during dispatch, and certainly not by turning around and asking the user a question the phone already knows the answer to.

On-device AI Agent refers to an intelligent agent whose inference and decision-making happen primarily on the user's device (phone, tablet, etc.) locally, rather than relying on cloud servers. Compared with cloud Agents, its core difference isn't model capability but data boundaries: sensitive information such as the user's location, calendar, and contacts never leaves the device, significantly reducing privacy risk. The cost is constrained local compute and memory—model size, context length, and inference speed are all limited. This directly determines the architectural orientation of on-device Agents: you can't simply copy the cloud paradigm of "a large model reads all the context, then plans." Instead, tasks must be broken into smaller steps with strong determinism. The context resolution and confidence tiering described in this article are essentially engineering rules that replace a large model's free-form reasoning under local compute constraints, trading for interpretability and low latency.

Context Resolution Before Dispatch

The team added a resolution step before every action dispatch called ContextResolver. It takes the raw intent and device context and attempts to fill in missing parameters before exposing anything to the user. For "my usual airport," for example, the resolver filters saved locations by category and takes the most frequently used one as the candidate address.

Resolution results fall into three paths: parameters resolved with high confidence are filled automatically; medium-confidence parameters are presented as pre-filled suggestions for the user to confirm or edit; only parameters that cannot be resolved at all are marked as missing and asked about.

The rule is stated plainly: never ask for information the phone already knows. It looks simple, but achieving it requires the resolver to have explicit handling logic for every parameter type (location, date/time, contacts, etc.), and to return an UnresolvedParameter rather than failing silently when it can't resolve something.

Confidence Tiering: Keeping the Agent from Confidently Doing the Wrong Thing

Not all resolution results are created equal. A saved location used 40 times is clearly more trustworthy than a place inferred from a calendar event three months ago.

StareBrain therefore defines four confidence levels: HIGH, MEDIUM, LOW, and UNRESOLVABLE. Each parameter value carries its confidence and source information (ContextSource), which determines behavior: high confidence is filled automatically and shown in the confirmation summary; medium confidence is pre-filled with a "Is this right?" prompt; low confidence or unresolvable goes straight to the user as a question.

The value of this tiering is balance: it reduces how much information users have to repeat, while preventing the Agent from making decisions on the user's behalf when uncertain. In the team's words, it keeps the Agent from "confidently doing the wrong thing."

Confidence Score in machine learning usually refers to the model's estimated probability that a prediction is correct. Traditional classification models output confidence as a continuous value (e.g., 0.87), but an on-device Agent faces a decision problem—"should I fill in this parameter for the user?"—so continuous scores are often discretized into tiers to enable explicit interaction policies. The benefit of discretization is predictable, testable behavior: each tier maps to a fixed action (auto-fill, pre-fill for confirmation, ask directly), with no behavior jitter near thresholds. The risk is that the manually set tier boundaries may be too conservative or too aggressive and need calibration from user feedback in beta testing. The article binds source information (ContextSource) to confidence precisely to make every decision traceable—so users or developers can answer "why does the Agent think this address is right?"

How a Hard Privacy Constraint Shapes the Architecture

The entire system runs on-device; no context ever leaves the phone. This isn't a preference—it's a hard constraint.

The direct consequence: the team cannot use a cloud model to reason over full device context, because full context isn't allowed to be uploaded at all. The resolution logic must be deterministic and lightweight enough to execute locally—which means it must explicitly state which signals it used and how it weighed them.

Compared with cloud approaches, this is a tighter constraint. But it's exactly what makes the privacy promise credible. The team's position is clear: the statement "your data never leaves your phone" only means something when the architecture actually enforces it.

On-device Inference refers to a model or rule engine running directly on the phone's SoC rather than sending input to a remote server. It relies on mobile inference frameworks (such as the neural network runtimes provided by device vendors) and model compression techniques like quantization and pruning to shrink models to a few hundred MB or less. The resolver in this article chooses deterministic rules over a large model partly for this reason: running a lightweight, auditable rule engine locally is more controllable than deploying and invoking a large model on the phone, requires no network, and responds faster. It's important to distinguish that "running locally" does not by itself equal "privacy-safe"—if local logs or caches aren't encrypted, data can still leak. So the real hard constraint also includes companion design choices: no disk persistence, no upload, no traces.

What It Solves, and What It Doesn't

Context resolution solves the "should already know" problem at the structured-data level—saved locations, calendar events, contacts. It does not solve open-ended inference (e.g., "what do I usually do here?"), nor does it solve cross-app state problems that require proactively reading app contents.

The team's assessment: those are harder problems, and the structured-data layer is the foundation. Get this layer right first—so users are no longer asked for information the phone already knows—and that's the prerequisite for any grander goal.

Their positioning is also worth noting: not to build the most capable Agent, but to build one that "doesn't make you feel like you're talking to something that doesn't understand you." In the on-device AI race, that standard may be closer to the essence of user experience than parameter count.

One-line Summary

StareBrain uses a local context resolution layer to let on-device Agents auto-fill information the phone already knows before dispatching actions.

Paragraph Summary

For on-device AI Agents, the StareBrain team proposes a context architecture that splits the Agent's needed context into action context and device context—the latter scattered across the OS and multiple apps, each with its own permission model. The core design has a ContextResolver resolve parameters before action dispatch, following the rule "never ask for information the phone already knows," and uses four confidence levels—HIGH, MEDIUM, LOW, UNRESOLVABLE—to decide between auto-fill, pre-fill for confirmation, or asking directly, keeping the Agent from confidently doing the wrong thing. Running entirely locally is a hard constraint that forces resolution logic to be deterministic and interpretable, and makes the privacy promise credible. The approach only addresses the structured-data layer; open-ended inference and cross-app state reading remain unsolved.