QuanLing
A quantitative linguistics framework based on pre-trained language models that measures distances between related languages using three metrics: sentence embedding distance, tokenization fragmentation rate, and masked language model mutual perplexity.
Timeline (last 90 days)
The QuanLing (Quantitative Linguistics via Pretrained Language Models) framework quantifies linguistic distances through pretrained language models.
The research team had previously applied the QuanLing framework to the North Germanic language family (Danish, Norwegian Bokmål, and Swedish) and completed preliminary validation.
QuanLing's methodology combines three indicators: sentence embedding distance, tokenization fragmentation rate, and masked language model mutual intelligibility.
The West Romance language experiment was conducted on four languages—French, Portuguese, Spanish, and Italian—using English as an anchor point and 150 sets of four-language parallel sentences as the data foundation.
Portuguese and Spanish are the closest pair among the four languages, with a LaBSE distance of only 0.0229.
French and Italian are the most distant pair, with a LaBSE distance of 0.0338.
The absolute distance range of the West Romance language family is 0.011, greater than the 0.008 of the North Germanic language family.
The relative ratio of the West Romance language family is 1.48, while that of the North Germanic language family is 1.67.
French achieved a top-1 accuracy of 36.12% in MLM prediction, while Italian was only 29.28%.
The study attributes French's higher MLM predictability to the characteristic of French orthography–phonology decoupling.
All Facts (10)
The QuanLing (Quantitative Linguistics via Pretrained Language Models) framework quantifies linguistic distances through pretrained language models.
50%UnverifiedThe research team had previously applied the QuanLing framework to the North Germanic language family (Danish, Norwegian Bokmål, and Swedish) and completed preliminary validation.
50%UnverifiedQuanLing's methodology combines three indicators: sentence embedding distance, tokenization fragmentation rate, and masked language model mutual intelligibility.
50%UnverifiedThe West Romance language experiment was conducted on four languages—French, Portuguese, Spanish, and Italian—using English as an anchor point and 150 sets of four-language parallel sentences as the data foundation.
50%UnverifiedPortuguese and Spanish are the closest pair among the four languages, with a LaBSE distance of only 0.0229.
50%UnverifiedFrench and Italian are the most distant pair, with a LaBSE distance of 0.0338.
50%UnverifiedThe absolute distance range of the West Romance language family is 0.011, greater than the 0.008 of the North Germanic language family.
50%UnverifiedThe relative ratio of the West Romance language family is 1.48, while that of the North Germanic language family is 1.67.
50%UnverifiedFrench achieved a top-1 accuracy of 36.12% in MLM prediction, while Italian was only 29.28%.
50%UnverifiedThe study attributes French's higher MLM predictability to the characteristic of French orthography–phonology decoupling.
50%