Recognize But Can't Generate: The Evaluation Blind Spot in LLMs for Cultural Kinship Terms

A Capability Gap Hidden by Multiple Choice

Do large language models (LLMs) truly understand the complex kinship terminology found across different cultures? Most prior research has examined this as a recognition task using multiple-choice questions. But a new arXiv paper raises a sharp challenge: choosing the right answer is not the same as being able to produce it independently.

A research team asked five open-weight models to generate kinship terms in three non-Western languages — Hindi, Tamil, and Korean — alongside a controlled experiment using the same relational prompts but presenting four candidate options for the model to choose from. The results revealed a striking gap.

90% vs. 36%: The Enormous Gap Between Selection and Generation

On the exact same "relationship–language" cells, GPT OSS 120B achieved a selection accuracy of 90.67% across 75 valid cells — but when required to generate the term independently, only 36.00% of attempts produced an accepted answer.

Llama 3.3 70B showed the exact same pattern: 77.92% selection accuracy versus just 24.24% generation accuracy.

What does this gap mean? The researchers interpret it with notable restraint. Because the multiple-choice format directly presents candidate terms and does not require the model to produce the target script itself, the gap should be understood as an evaluation format gap rather than direct evidence that the model's lexical knowledge is fully intact. In other words, we cannot simply say "the model knows but is just too lazy to write it" — generation tasks genuinely test a different dimension of capability.

This "evaluation format gap" has a grounding in cognitive science: recognition and recall are two distinct modes of memory retrieval. Recognition tasks (such as multiple choice) only require identifying the correct answer upon seeing it, while recall tasks (such as generation) demand active retrieval and output from internal knowledge. Classical psychology experiments have long established that humans systematically outperform on recognition versus recall tasks. A similar dissociation in LLMs is not surprising — but the magnitude of the gap, exceeding 50 percentage points, reveals a problem with serious implications for real-world deployment: when users need a model to directly "write out" a kinship term, its reliability is far lower than benchmark scores suggest.

Even With Explicit Relationships, Models Still Struggle

The study also included a more compelling condition: prompts that explicitly specified the target language (L3 prompts), ruling out the possibility that models simply misunderstood the question.

Even so, model performance varied dramatically — GLM-5.1 reached 72.29%, while Llama-3.3 70B managed only 24.24%. This shows that even when a relationship is clearly stated, generating culturally specific kinship terms remains a difficult task. This finding is especially significant because it rules out "comprehension bias" and pins the problem squarely on generation capability itself.

Paternal-Lineage Advantage Is Language-Specific

One interesting finding in the paper concerns the "paternal-lineage advantage" — whether models perform better on kinship terms from the paternal side of the family.

The results show this advantage is language-specific: it is pronounced in Hindi but weak or even reversed in Korean. Tamil's "shared-term pairs" were used as a control to help account for measurement error.

This finding reminds us that kinship terminology systems are deeply embedded in their respective cultural and linguistic structures. Hindi's paternal kinship vocabulary is highly differentiated, while Korean has its own distinct honorific and kinship classification logic — model performance cannot be generalized across cultures.

Understanding this finding requires appreciating the fundamental differences among the three languages' kinship systems. Hindi, an Indo-European language, preserves fine-grained paternal/maternal kinship distinctions (e.g., a father's elder brother "taau" versus a mother's brother "maama" are entirely different words), with the vocabulary itself encoding a paternal-priority cultural logic. Korean features an honorific kinship system centered on the speaker's gender and relative seniority (e.g., "오빠/형"), with a different logic for distinguishing paternal and maternal kin — some terms are even shared across both sides. Tamil, as a representative Dravidian language, has a kinship system rooted in cross-cousin marriage traditions, featuring many shared terms applicable to both paternal and maternal relatives — which is precisely why researchers used it as a baseline for controlling measurement error. Differences in the volume of training data and cultural knowledge density across these three languages, compounded by the structural complexity of each kinship system, jointly explain the language-specific patterns in model performance.

Implications for Evaluation Methodology

The most central contribution of this research is its critical reflection on current paradigms for evaluating multilingual LLM capabilities.

For a long time, multiple-choice questions have dominated because they are easy to quantify and automate. But the data in this paper make it clear: multiple-choice formats systematically overestimate models' actual generation capabilities. A model that scores 90 on a selection task may only perform at a 36-point level when it actually has to speak.

The authors therefore advocate introducing generation-based evaluation alongside multiple-choice testing. This is especially critical for low-resource languages and culturally specific knowledge — because real-world applications require models to produce the correct content, not merely recognize it.

For researchers and product teams concerned with the genuine usability of models in non-English, non-Western cultural contexts, this is a cautionary signal: strong benchmark scores don't necessarily mean the model is actually usable.

Low-resource languages refer to languages with limited training data and underdeveloped NLP toolchains, typically contrasted with high-resource languages like English and Chinese. Hindi, Tamil, and Korean all occupy a far smaller share of global internet corpora than English, though the gaps among the three are also significant — Hindi has relatively more digitized text, while Tamil has lower web corpus density. This context makes the study's findings all the more sobering: a model's ability to generate culturally specific vocabulary is constrained not only by the sheer volume of language resources, but also by whether that vocabulary appears in training data with sufficient structured context. A kinship term may appear many times in training data, yet if it always appears in passive recognition contexts rather than active production contexts, the model will still struggle to develop reliable generation capability.