Inferring L1 Backgrounds from L2 English via LLM Token Probabilities: 70B Model Achieves SOTA

Mining Native Language Clues from Token Probabilities

Native Language Identification (NLI) aims to infer a speaker's first language (L1) by analyzing their second language (L2) output. Traditional approaches mostly rely on handcrafted features or fine-tuned classifiers, but a new study proposes a scoring method based on LLM token probabilities: each text sample is scored as a continuation of a prompt prefix naming a candidate L1, and the highest-scoring candidate L1 is predicted. The method uses Llama 3.1 8B and 70B models, is evaluated on three datasets—TOEFL9, Write and Improve 2024, and BULATS-L1—and is compared with prior prompt-based classification approaches.

Method Core: Prompt Prefixes and Probability Scoring

The research team designed a simple yet effective pipeline. For each L2 English text to be classified, they construct a prompt prefix that explicitly names a candidate L1 (e.g., "This text was written by a speaker whose first language is Chinese."), then compute the token probability score of the text as a continuation of that prefix. This is repeated for all candidate L1s, and the highest-scoring one is chosen as the prediction. The method requires no additional training and relies solely on the inherent knowledge of pretrained LLMs.

Token probability scoring: In autoregressive language models, given a prefix sequence, the model outputs a probability distribution over the next token. When scoring a text as a continuation of a prefix, one typically sums the log probabilities of each token in the text conditioned on the preceding context (including the prefix), then normalizes by the number of tokens to obtain the average log-likelihood. This score reflects how well the text "matches" the given prefix in the model's view. In this method, prefixes corresponding to different candidate L1s steer the model toward different probability distributions, so the score differences of the same text under different prefixes constitute the signal for distinguishing L1 backgrounds.

Direct prompt classification: This refers to having the LLM read the text directly and output an L1 category label, typically relying on instruction-tuned models to follow formatting requirements. This approach is limited by the model's instruction-following ability, the stability of output parsing, and how candidate categories are presented in the prompt; probability scoring bypasses generation and parsing and directly exploits the model's internal distributional information, making it often more robust on smaller models.

Experimental Setup and Datasets

The experiments used Llama 3.1 8B and 70B versions on three public datasets: TOEFL9 (TOEFL essays with 9 L1 backgrounds), Write and Improve 2024 (large-scale learner English essays), and BULATS-L1 (Business Language Testing Service writing). These datasets cover L2 English output at different proficiency levels and in different scenarios, providing a basis for evaluating the method's generalization ability. The study compared the performance of token probability scoring versus direct prompt classification (i.e., having the LLM directly output the L1 category).

TOEFL9: This dataset contains TOEFL essays written by test-takers with diverse native language backgrounds, typically covering 9 L1s (e.g., Chinese, Arabic, Spanish, etc.), and is one of the earlier and commonly used benchmarks in NLI research. Its text lengths and writing scenarios are relatively uniform, making it easy to control variables.

Write and Improve 2024: Learner English essays from the University of Cambridge's Write & Improve online writing platform. It is large in scale with diverse L1 backgrounds, and its text proficiency and topic distribution are closer to real learning scenarios, making it commonly used to test generalization on large-scale heterogeneous data.

BULATS-L1: Writing samples based on the BULATS business English exam, oriented toward workplace and business communication. It complements academic writing datasets and allows evaluation of the method's performance across different registers.

Llama 3.1: Meta's open-source large language model series, where 8B and 70B refer to parameter scales. The study uses its pretrained/instruct versions without fine-tuning, so performance differences mainly reflect model scale and the pretrained knowledge's ability to encode L1-related linguistic features.

Results: Token Probabilities Significantly Outperform Direct Classification

Experimental results show that the token probability scoring method substantially outperforms direct classification, with particularly notable gains on the 8B model. Direct classification may be limited by the LLM's instruction-following ability and output format constraints, whereas probability scoring more fully exploits the model's internal distributional information. More strikingly, the 70B model achieves state-of-the-art (SOTA) performance on all datasets, surpassing prior prompt-based methods. This finding indicates that the token probabilities of pretrained LLMs encode subtle linguistic features related to L1 background.

Impact of Model Scale

The significant gains on the 8B model and the SOTA performance of the 70B model together reveal the role of model scale. Smaller models may lack sufficient linguistic knowledge for direct classification, but their token probabilities can still capture discriminative signals; larger models further translate these signals into more accurate predictions. This offers a feasible approach for using smaller models in resource-constrained settings—improving performance through probability scoring rather than direct classification.

Implications and Limitations

This study provides a new lightweight paradigm for the NLI task: competitive performance can be achieved without fine-tuning, merely by designing prompt prefixes and computing probabilities. This lowers the barrier to applying NLI and demonstrates the potential of LLMs in linguistic profiling. However, the method relies on enumerating candidate L1s, and the computational cost of probability scoring grows linearly with the number of candidates. Future work could explore more efficient scoring strategies and cross-lingual generalization.

Practical Application Prospects

From educational technology to security forensics, L1 background inference has its uses. For example, language learning platforms could use it to provide personalized feedback to learners, and content moderation systems could help identify linguistic traces in disinformation. However, privacy and ethical issues must be considered to prevent misuse of linguistic profiling technology.

In summary, this study confirms the value of pretrained LLM token probabilities in L1 background identification, bringing a simple yet powerful new tool to the NLI field.

One-line Summary

No fine-tuning needed: LLM token probability scoring infers the native language of L2 English writers, with the 70B model achieving SOTA.

Paragraph Summary

This study proposes an NLI method based on LLM token probabilities: L2 English texts are scored as continuations of prompt prefixes naming candidate L1s, and the highest-scoring candidate is predicted as the L1. Using Llama 3.1 8B and 70B, it is evaluated on TOEFL9, Write and Improve 2024, and BULATS-L1. Results show that token probability scoring significantly outperforms direct prompt classification, especially on the 8B model; the 70B model achieves SOTA on all datasets. This indicates that pretrained LLM token probabilities encode subtle linguistic features that distinguish L1 backgrounds, offering a lightweight, fine-tuning-free paradigm for NLI—though the method relies on enumerating candidate L1s, and computational cost grows linearly with the number of candidates.