Tracking Sanskrit Semantic Change with Computational Methods: NLP Challenges for Low-Resource Ancient Languages

Diachronic word embeddings have become the modern standard tool for tracking semantic change, but their effectiveness has largely been validated on modern, high-resource languages with clear word boundaries. What happens when researchers try to apply this paradigm to an ancient, resource-scarce language like Sanskrit? A newly published arXiv paper provides a systematic answer.

Why Sanskrit Is Such a Hard Nut to Crack

The challenges Sanskrit poses for computational linguistics are fundamentally different from those of modern languages. It has several thorny linguistic features: phonological fusion caused by sandhi, complex morphological inflection, extensive compounding, and pervasive polysemy.

These features stack on top of each other, making even the most basic task — identifying word boundaries — extremely difficult. In a language like English, spaces naturally segment words, and word embedding models can directly consume token sequences. In Sanskrit text, however, words frequently merge into continuous character streams in writing, leaving models unable to determine where one "word" begins and another ends. This is the core reason why the diachronic word embedding paradigm cannot be straightforwardly applied to Sanskrit.

Sandhi is a phonological rule in Sanskrit where adjacent phonemes merge and transform at word boundaries. For example, when two words are adjacent, a final vowel and an initial vowel may merge into a new vowel, or even alter consonant forms. This rule is faithfully recorded in writing, causing sentences to appear visually as nearly unbroken streams of characters with few spaces. For NLP models, sandhi splitting is the very first hurdle in Sanskrit analysis — errors at this stage contaminate all subsequent word-level statistics. Additionally, Sanskrit's morphological inflection system is extraordinarily rich; a single root can give rise to dozens of inflected forms. Without lemmatization to collapse all inflected forms into a single lexeme, sparse word frequency statistics severely degrade embedding quality. This is precisely why preprocessing occupies a far more central role in Sanskrit NLP than in processing English.

From Raw Corpus to Trainable Data

The authors built a corpus of 2.7 million tokens spanning four classical periods of Sanskrit literature, providing the temporal breadth necessary for "diachronic" analysis. Cross-period data is what makes it possible to observe how a word's meaning drifts over time.

The most critical step in the processing pipeline was solving the word boundary problem described above. The authors employed a byte-level neural sandhi splitter and lemmatizer to restore the fused character streams into individual lexemes, then trained word embeddings separately for each period on this foundation. This "split first, then model" approach essentially supplies the preprocessing conditions that ancient low-resource languages lack but modern languages naturally possess.

The core idea behind diachronic word embeddings is to train independent vector spaces on corpora from different historical periods, then use alignment techniques to map each period's vectors into a shared coordinate system, enabling direct comparison of the same word's vectors across time. A common alignment method is orthogonal Procrustes transformation, which minimizes the distance between aligned word pairs while preserving the internal geometric structure of each period. Changes in cosine distance between vectors are then treated as a quantitative measure of semantic drift. This framework has been extensively validated on languages like English and German, but its underlying assumptions — sufficiently large corpora, clear word boundaries, relatively stable spelling — are rarely met in classical languages. This is the fundamental reason why this study requires substantial additional engineering at the preprocessing level.

How to Validate the Results

A major challenge in semantic change research is evaluation — how do you know whether the drift a model captures is real rather than training noise? The authors' approach is worth emulating: they manually compiled a validation set from historical scholarship, using semantic changes already documented by linguists as a "gold standard."

Evaluation used a directional test based on anchor displacement — checking whether the direction of semantic drift indicated by the model aligns with the direction established by philological scholarship. The results are compelling: of 21 testable semantic changes, 19 had movement directions consistent with philological evidence. The authors ran a sign test on this result, obtaining a significance level of p=0.00011, meaning this degree of agreement is extremely unlikely to have occurred by chance.

The sign test is a non-parametric statistical method used to determine whether a binary outcome (such as "direction correct" or "direction incorrect") occurs at a frequency that significantly deviates from a random baseline. In this study's context, if the model's judgments about the direction of semantic drift were completely random, each of the 21 test cases would have a 0.5 probability of being correct. With 19 observed correct predictions, the sign test calculates the probability — under the null hypothesis — of obtaining "at least 19 correct," yielding p=0.00011. This extremely low p-value indicates that the model's directional judgments are highly consistent with philological evidence and are almost certainly not coincidental. Compared to simply reporting cosine similarity values, directional validation carries greater linguistic meaning, because it directly corresponds to the interpretable historical question of "whether word meaning evolved in a known direction."

The Significance and Limitations of Paradigm Transfer

The value of this work lies not only in the conclusion that "semantic change analysis is possible for Sanskrit," but in what it reveals about the specific constraints involved in transferring a paradigm. The authors explicitly identify which configurations Sanskrit as a language "forced" upon them — in other words, the language's properties directly determined which modeling approaches were viable and which were not, rather than being arbitrary hyperparameter choices.

For researchers working on low-resource language NLP, this provides a complete, replicable pipeline: corpus construction → neural splitting and lemmatization → period-specific training → directional validation against historical scholarship. The authors also honestly identify areas for future improvement, making clear the current system is far from a final solution.

Implications for Low-Resource Language NLP

Mainstream NLP research has long concentrated on high-resource languages like English, while classical languages and minority languages are frequently marginalized. The significance of this paper is that it demonstrates how mature modern methods, when thoughtfully adapted, can genuinely serve classic questions in the humanities — such as the semantic evolution of Sanskrit literature.

More importantly, it models a rigorous approach to evaluation: rather than settling for visually appealing results, it returns to the philological tradition to find verifiable ground truth. In the age of large language models, this practice of "using domain knowledge to constrain and validate model outputs" carries universal reference value for any effort to apply AI to specialized fields.