Microsoft ATLAS Framework: Rethinking AI Design and Evaluation Through Cultural Infrastructure

When AI Meets Culture: Why General-Purpose Models Fail in the Field

A field study from Microsoft Research reveals a key contradiction: AI systems perform excellently in English, high-resource environments, but when they enter agricultural communities in Kenya and India, farmers often abandon them after just a few conversations. The problem isn't model capability—it's that cultural context is systematically ignored.

In her lecture, Dr. Liz Ankrah, a research scientist at Microsoft Research, posed a core question: What happens when AI quality depends on knowledge that technical systems cannot easily capture? This knowledge includes local language practices, social relationships, practical resource constraints, and how community history shapes people's trust in and judgment of AI.

Her answer is ATLAS—a practical playbook grounded in ethnography and participatory research that helps teams identify cultural assumptions throughout the AI lifecycle and use community knowledge to guide design, development, deployment, and evaluation.

Culture as Infrastructure: An Overlooked Analytical Framework

Ankrah defines culture as the "infrastructure" of AI. In a typical AI interaction, the user expresses a need, the AI generates a response, and the user judges and takes action—culture plays a key role in all three steps: shared language helps users articulate problems, knowledge relationships help users judge responses, and local practices determine whether a response can translate into real action.

Social science definitions of culture are complex and varied—from multi-level models spanning nations to individuals, to dimensions of individualism versus collectivism, to practice-based constructivism. ATLAS's contribution lies in proposing an operational four-dimensional cultural analysis framework:

• Contextual: What is the domain, task, modality, and use scenario?
• Relational: How do people connect with others and with systems? What are existing knowledge practices?
• Consequential: What decisions, actions, and outcomes does system use produce?
• Dynamic: Relationships evolve; today's consequences are not tomorrow's consequences.

The framework is designed to make culture analytically "usable" while preserving its complexity and variability, rather than reducing culture to a few parameterizable labels.

Ethnography is a qualitative research method that emphasizes researchers' long-term immersion in a specific community, using participatory observation, in-depth interviews, and everyday interaction to understand local people's meaning worlds and practical logic. It does not presuppose a theoretical framework but instead generates insights "bottom-up" from the field, and is especially good at capturing tacit knowledge that questionnaires or metrics struggle to quantify—such as how farmers judge whether a piece of advice is credible, how information flows through a community, and which social relationships determine the feasibility of action. In AI design, the value of ethnography lies in revealing cultural assumptions that system developers take for granted, avoiding the default treatment of interaction patterns from high-resource environments as universal standards.

Project Gecko: Three Key Findings from the Field

ATLAS's empirical foundation comes from Project Gecko—a multi-stage field study conducted in agricultural communities in Kenya and India. The research team partnered with Digital Green to conduct interviews, farm visits, onboarding observations, prototype workshops, and co-design sessions across multiple sites including Nandi, Nyeri, and Bihar, involving different languages, knowledge systems, role relationships, and seasonal cycles.

Despite enormous contextual differences, three findings held across all cases:

Practices and Relationships Determine the Usability of Advice

AI never enters a blank knowledge vacuum. Farmers already have an information ecosystem composed of peer networks, extension officers, and other digital tools. When AI is added, farmers need to reinterpret and translate AI outputs, then bring them back to the community for discussion before action becomes possible. This means "feasible action" is far more than the AI giving an answer—local resource conditions, timing, and responsibility allocation are often things the system cannot capture.

High-resource and low-resource environments are a common distinction in natural language processing: the former refers to contexts with large amounts of annotated data, mature evaluation benchmarks, and rich language resources (such as the English internet), while the latter refers to contexts with scarce data, many language variants, and weak digital infrastructure (such as Kikuyu in Kenya or Bihari dialects in India). This distinction concerns not only data volume but also the form of knowledge—much knowledge in low-resource environments exists in oral, practical, and community-relationship forms that are difficult to capture in text corpora. The "failure" of general-purpose models in these environments is often not a failure of language understanding but a disconnect between the cultural scripts implicit in their training data and local practices.

Topic Matching Is Not Task Usability

Conversation analysis revealed a typical pattern: a farmer asks "How much does it cost to raise a pig," and the AI recognizes the topic and begins discussing pig housing, breeding stock, and cost factors, but completely fails to provide the usable price information the farmer actually needs to make a decision. The answer is on-topic but unusable. This exposes a systematic mismatch between machine and human definitions of "useful."

Ankrah therefore proposed directions for interface redesign: making context visible, allowing clarifying follow-ups, presenting sources and limitations when giving answers, and making explicit when a task should be handed back to humans. In co-design workshops, farmers spontaneously filled in details for each other—"You have to specify it's one cow, 300 kilograms, and how much to feed"—this practice of collaborative questioning suggests how AI interaction could support multiple users jointly constructing a problem.

Evaluation Is Social, Not Just About Accuracy

Farmers judge AI responses in ways far more complex than accuracy: Does the response match my existing knowledge and beliefs? Do others in the community approve of this answer? Farmers weight farmer-to-farmer trust far above any institutional knowledge. Completeness is subjective—the pig-farming answer, though rich in detail, was judged "incomplete" because it lacked information farmers considered necessary. Usefulness is personal—some want less text, some want charts, some want specific agricultural context.

This directly leads to a research hypothesis: AI quality depends on the degree to which user needs and goals are met, and must therefore be evaluated using human-centered values. The team has a paper on this currently under review.

Human-Centered Evaluation advocates shifting the evaluation focus from system-internal metrics (such as accuracy, F1 scores) to the degree to which users' actual goals are achieved. It typically combines qualitative and quantitative methods: identifying users' real needs through task analysis, observing system performance in real usage scenarios through contextualized testing, and involving stakeholders in defining what counts as "good." In cross-cultural AI scenarios, this approach is especially important because "useful" is itself a local concept—the same answer may be judged complete or incomplete, credible or not, in different communities, and these judgments cannot be captured by general benchmarks.

How to Use the ATLAS Playbook: From Assumptions to Decisions

ATLAS organizes the workflow around four questions: What must be understood in context? Whose knowledge and relationships influence use? What consequences will arise? How can the system adapt to change and revisit assumptions?

The playbook provides concrete templates that guide teams in making implicit assumptions explicit. For example, if the current system assumes "people know what they want to ask," the team can investigate how people actually express needs and what kinds of answers they find satisfying. Local investigation might find: people construct problems collectively, key details emerge during conversation, and they consult others before acting. Based on this, the team can broaden user personas (from individual farmers to farmers and their families), design interactions that support contextual elaboration, and present community-endorsed sources in AI guidance.

The playbook offers different pathways depending on where a team is: when no system exists yet, it focuses on contextual understanding and co-creation; when a system already exists, it focuses on adaptation and repair.

Open Questions: Intent Capture and Community-Centered AI

During the Q&A, Ankrah and colleagues acknowledged several open questions. Intent capture is the foremost challenge: how can AI obtain sufficient context during conversation rather than assuming one sentence is enough? "Alignment," in her view, is not just a technical problem but a process of reaching shared definitions through dialogue—just as "fairness" means different things to different people.

Interaction modalities are also being explored: from pure text to voice (the team is developing speech models for Kenyan languages such as Kikuyu and Swahili, because farmers prefer hands-free operation), to video and image interaction—what output forms do farmers actually need?

Another gap specifically emphasized: current AI systems are almost all designed for "individual-to-individual" interaction, but in project contexts, farmers take answers back to community groups for discussion and a "sanity check." How to design community-centered AI—enabling cooperatives or women's groups to interact with AI as a collective and fine-tune its contextual understanding—is what the team sees as the most promising direction. In her colleague's words, "We can't just copy the Western model of one person facing one AI tool."

Ankrah also mentioned a thought-provoking observation: research suggests AI agrees with users about 49% of the time. For farmers, a system that agrees with you even when it's wrong doesn't help you achieve your goals. True conversational AI should, like human conversation, mutually correct understanding through interaction—not just nod politely.

AI Alignment typically refers to technical research aimed at making AI system behavior consistent with human intentions and values, with mainstream approaches including reinforcement learning from human feedback (RLHF). But Ankrah's perspective points to a more fundamental dimension: alignment is not just technical parameter tuning but a social negotiation process—different communities define "fairness," "usefulness," and "respect" differently, and alignment goals themselves need to be jointly defined through ongoing dialogue with users. In cross-cultural contexts, if alignment goals are unilaterally set by high-resource environments, systems may be "aligned" on technical metrics yet fail in local practice.