NepKANUN: A Deep Dive into Nepal's RAG-Based Legal AI Assistant

An AI Experiment for Low-Resource Legal Domains

When it comes to accessing legal information, the gap created by language and geography is often overlooked by the tech community. Most legal AI tools are built around dominant languages like English, while regions like Nepal — where legal terminology is complex, resources are limited, and misinformation is rampant — present an extremely high barrier for ordinary citizens trying to understand their legal rights.

A new paper on arXiv introduces NepKANUN, an AI legal assistant designed specifically for Nepali legal texts. It is built on a fine-tuned large language model and integrates a Retrieval-Augmented Generation (RAG) framework, with the goal of democratizing access to legal knowledge.

How RAG Addresses Accuracy Challenges in Legal Q&A

Legal question-answering demands an extremely high level of accuracy — even a slight error can mislead users. Relying purely on a generative model risks "hallucinations," where the model fabricates plausible-sounding but incorrect legal provisions. NepKANUN's approach combines generation with retrieval: when a user submits a legal query in natural language, the system first retrieves relevant content from a reliable legal corpus, then uses the fine-tuned language model to generate a precise, concise answer.

This "retrieve-then-generate" combination has become the dominant paradigm for professional-domain AI applications. The retrieval step ensures answers are grounded in real sources, reducing fabrication; the generation step transforms dense, opaque legal language into plain, accessible explanations. For a domain like law, where the cost of errors is extremely high, RAG's value is especially pronounced.

RAG (Retrieval-Augmented Generation) typically operates in three phases: First, the legal corpus (statutes, case law, full regulatory texts) is chunked into text segments and encoded into high-dimensional vectors via an embedding model, then stored in a vector database. Second, when a user asks a question, it is similarly encoded into a vector and the most relevant text segments are retrieved via similarity search (e.g., cosine similarity). Finally, the retrieved results are concatenated as "context" into the prompt, which the language model uses to generate the final answer. The key advantage of this architecture is "externalized knowledge" — the model's parameters don't need to store all specialized knowledge; instead, the external knowledge base is dynamically queried at inference time. This reduces hallucinations and makes knowledge updates easy (just update the corpus rather than retrain the model). For law, a domain where knowledge is highly time-sensitive and errors carry serious consequences, RAG has become almost the default solution.

Training Data and Performance

The research team used a custom, high-quality question-answer pair dataset to train the model. Data quality is decisive for the effectiveness of vertical-domain AI, and in a low-resource language legal setting without publicly available corpora, constructing a custom dataset is itself a significant contribution.

For evaluation, the team used BERTScore as the primary metric and tested performance across difficulty levels. Results showed:

• Simple questions: F1 score of 0.82
• Medium-difficulty questions: F1 score of 0.77
• Complex questions: F1 score of 0.71

As expected, performance declines as question complexity increases — more complex legal questions often involve cross-referencing multiple provisions and multi-step contextual reasoning, which is more demanding for both retrieval and generation. The 0.71 score on complex questions indicates room for improvement in high-difficulty scenarios, but overall performance is already at a practically usable level.

Beyond automated metrics, the study also validated the system's practical utility through expert review, providing an additional layer of credibility for real-world deployment.

BERTScore is a text similarity evaluation metric based on pre-trained language models (such as BERT). Unlike traditional metrics like BLEU or ROUGE, which rely on lexical overlap, BERTScore measures quality by computing the cosine similarity between candidate and reference texts in semantic embedding space, making it more tolerant of paraphrasing. BERTScore calculates Precision, Recall, and F1 separately, with F1 commonly used as the overall performance indicator. The advantage of using BERTScore for legal Q&A evaluation is that legal answers often have multiple valid phrasings — semantic matching more accurately reflects actual correctness than surface-level matching. That said, BERTScore has its limitations: it measures expressive similarity rather than factual accuracy, which is precisely why the research team supplemented it with expert review as an additional validation method.

Implications for Low-Resource Language AI Deployment

The significance of NepKANUN extends beyond being a Nepali legal tool — it demonstrates a replicable path: even in low-resource, niche-language vertical domains, a practical professional AI assistant can be built using a custom dataset paired with a RAG framework.

This holds reference value for countless communities speaking non-dominant languages worldwide. Public service domains like law, healthcare, and government services universally suffer from information asymmetry, and targeted AI solutions have the potential to reduce these barriers and make technology genuinely useful for ordinary people's everyday needs.

Of course, for such systems to be deployed in actual practice, continuous refinement is still needed in areas like data coverage breadth, handling of complex queries, and accountability boundaries (AI-generated advice cannot replace the opinion of a qualified lawyer). But as an exploratory effort, NepKANUN provides a noteworthy model worth paying attention to.

In natural language processing, a "low-resource language" refers to a language lacking large-scale annotated corpora, pre-trained models, and benchmark datasets — Nepali falls into this category. Compared to high-resource languages like English, the core challenges for low-resource languages include: far less text data available for pre-training, near-total absence of domain-specific (e.g., legal, medical) parallel corpora, and significantly weaker performance from existing multilingual models (such as mBERT and XLM-R) on these languages. NepKANUN's approach — building a vertical-domain Q&A dataset from scratch combined with targeted fine-tuning — represents a pragmatic strategy for low-resource scenarios: rather than waiting for general-purpose large models to cover all languages, a modest annotation investment can yield usable performance on specific tasks. This way of thinking carries reference value for the vast majority of the world's roughly 7,000 languages that have yet to be covered by mainstream AI tools.