Microsoft Project Gecko: Using AI to Bridge Language Gaps for African Farmers

In the fields of Africa, language barriers often become the biggest obstacle for farmers seeking agricultural knowledge. The Microsoft Research Africa team, in collaboration with Digital Green, is attempting to break down this barrier through Project Gecko. Their PharmaChat app allows farmers to ask questions in their native language and receive precise answers based on video clips.

From 10,000 Videos to Intelligent Q&A

The core of Project Gecko is a simple yet powerful idea: provide farmers with the agricultural information they truly need in the language and cultural context they are familiar with. According to the project team, they have accumulated over 10,000 agricultural videos over the years. These videos are a valuable knowledge base, but traditionally it has been difficult for farmers to quickly find the specific clips they need.

PharmaChat changes all that. It is an AI-driven app where farmers can ask questions in their native language (such as Swahili), and the system returns the exact segment of the video that actually answers the question. The team is fine-tuning small language models to ensure answers are not only accurate but also easy for farmers to apply in their actual farming practices.

Small Language Models (SLMs) are the counterpart to large models like GPT-4, typically with fewer than a few billion parameters. After instruction fine-tuning and distillation on domain-specific data, they can run on edge devices or in low-bandwidth environments, with inference costs far lower than calling cloud-based large models. For rural African scenarios, the advantages of SLMs are particularly evident: they do not rely on a continuous, stable high-speed network and are easier to customize with local language corpora. PharmaChat, by fine-tuning an SLM, compresses domain knowledge (agricultural Q&A) into a locally deployable model, thereby reducing infrastructure requirements while ensuring answer accuracy.

Retrieval-Augmented Generation (RAG) is the key technical paradigm behind applications like PharmaChat. It does not require the model to memorize all knowledge; instead, it first retrieves the most relevant segments from an external knowledge base (in this case, over 10,000 agricultural videos) based on the user's question, then either hands them to the model for generation or directly returns the segment. This approach significantly reduces the risk of the model "hallucinating" and makes answers traceable—farmers see the actual footage from the video that explains the issue.

Into the Field: Testing AI in Real Scenarios

To build AI that truly serves everyone, you must go into the field. The project team conducted field tests in Kenya to observe how farmers received the app, what questions they initially asked, and what challenges they encountered during use. They also paid special attention to whether farmers preferred Swahili or English.

One farmer user reported: "Since we got PharmaChat, we've been using it to ask questions, and things are much better than before. It helped me recognize the mistakes I was making in my farming." Such voices from the front line validate the value of AI in agriculture.

MMCT Agent: The Evolution of Multimodal Answers

The project did not stop at video retrieval. The team added a powerful new feature to PharmaChat—the MMCT Agent, based on an intelligent video understanding system. Now PharmaChat can provide multimodal answers that are culturally and linguistically relevant to farmers.

Specifically, the system not only returns video clips but can also provide specific chapters from community farming books. This combination of text and video makes answers more complete. As one farmer put it: "Some people can't understand text when learning, but they can understand more by watching videos."

Multimodal retrieval refers to systems that process multiple forms of information simultaneously, such as text, images, video, and speech. The MMCT Agent needs to convert the farmer's voice or text question into a unified semantic representation, then perform cross-modal matching across multiple libraries including video frames, subtitle text, and book chapters. For example, if a farmer asks "What should I do about yellowing corn leaves?", the system must find both the video segment showing the yellowing symptom and the corresponding disease control chapter in the book. This is far more complex than pure text retrieval, because videos must first be segmented, transcribed, and visually featurized before they can be retrieved.

Performance Evaluation and the Kikuyu Language Model

The team is working with machine learning engineers to evaluate how these models perform across different modalities: text, image, video, and speech—which works better? What are the model's failure modes? How can accuracy be improved? The answers to these questions will determine whether AI can truly and reliably serve farmers.

Another important direction is creating a Kikuyu language model. For farmers who understand neither Swahili nor English, they can still ask questions and receive answers in the language they fully command. This reflects the project's commitment to linguistic diversity.

Low-resource languages refer to languages with very little data on the internet and in public corpora, and Kikuyu is one such language. Mainstream large models typically have poor support for such languages because they are almost absent from training data. Building a model for Kikuyu faces a double challenge: first, the lack of parallel corpora (Kikuyu-English aligned data), and second, the lack of evaluation benchmarks. Common approaches include cross-lingual transfer using related languages such as Swahili, crowdsourcing speech and text data from the community, and parameter-efficient fine-tuning (such as LoRA) to adapt models on limited data. This is also the technical meaning behind the project's emphasis on a "commitment to linguistic diversity."

Three Interactive Manuals: Empowering Others

The Project Gecko team is not content with building AI themselves; they want to help others do the same. To this end, they have released three interactive manuals: Paza, Vibhasha, and Atlas.

The Paza speech manual provides best practices for robust model training, deployment, and evaluation. Vibhasha offers practical guidance for building large language model applications across languages and cultures. Atlas guides developers in building AI based on real-life experience. These resources transform the project's experience into reusable knowledge, lowering the barrier for other teams to enter this field.

From field testing to open-source manuals, Project Gecko demonstrates a path to making AI truly inclusive for all—not replacing local knowledge with technology, but using technology to make local knowledge more accessible.

One-line Summary

Microsoft Project Gecko uses native-language AI Q&A to help African farmers precisely access farming knowledge from 10,000 videos.

Paragraph Summary

Microsoft Research Africa, in collaboration with Digital Green, is conducting Project Gecko, which developed the PharmaChat app that lets African farmers ask questions in their native language and receive answers in the form of video clips. The project builds a knowledge base from over 10,000 agricultural videos and fine-tunes small language models to ensure answers are accurate and actionable. The team conducted field tests in Kenya to observe real usage patterns and language preferences. They then added the MMCT Agent to enable multimodal answers that can return both video clips and chapters from farming books. The team is also evaluating performance across text, image, video, and speech modalities, and developing a Kikuyu language model to reach farmers who understand neither Swahili nor English. Finally, the project released three interactive manuals—Paza, Vibhasha, and Atlas—to turn experience into reusable resources and lower the barrier for others to build inclusive AI.