Georgian Language Benchmark: A Hard-Core Showdown Between OpenAI and Anthropic

A Georgian developer used one of the world's grammatically most complex languages to challenge eight models from OpenAI and Anthropic. The results reveal a widely overlooked truth about large language models: understanding a low-resource language and correctly generating that language are two entirely different things.

Why Use Georgian as a Test

Georgian is one of the oldest languages in the world, with its own independent writing system that belongs to no major language family. Its consonant clusters are strikingly dense — the author's example, გვფრცქვნი (meaning "you peel our skin"), strings together eight consecutive consonants in a single word. Its grammar runs even deeper: a single verb like მიყვარხარ can express "I love you" while simultaneously encoding agent, patient, and tense.

The author is a native Georgian speaker and language enthusiast, yet candidly admits that even most Georgians cannot use the language without grammatical errors. This makes it an ideal test for evaluating generative capability in large models — it is severely underrepresented in training corpora, and its rules are highly irregular. The entire dataset was hand-written from scratch by the author, with no machine translation or generated text, and was reviewed by a linguistics expert.

The reason Georgian verb morphology is particularly difficult to generate lies in its use of polypersonal agreement: a single verb form must simultaneously encode the person and number of the subject, direct object, and indirect object, plus tense, aspect, and mood — producing thousands of possible conjugations. Georgian also uses a unique system of screeves, tightly binding tense and aspect rather than treating them separately as Indo-European languages do. This means models cannot simply "look up" and swap out endings; they must understand the topological relationships between semantic roles to produce correct forms. The extreme scarcity of training data (roughly three to four orders of magnitude less than English) combined with the inherent irregularity of the rules makes Georgian a near-ideal language for testing the upper limits of generative capability.

Five Tasks: From Reading to Generation

Built on the Kaggle Benchmarks platform, the author designed tests across five dimensions, with each model answering a total of 287 scored questions:

Task Design

• Numbers (158 questions): Georgian uses a vigesimal (base-20) counting system, where 99 is expressed as "four twenties plus nineteen." The task also covers ordinals, fractions, decimals, and large numbers up to 10³⁹, with bidirectional conversion (digits to words, words to digits).
• Verbs (50 questions): Identifying "who did what to whom" within a verb form, constructing correct verb morphology, and selecting the correct prefix.
• Translation (12 passages): Georgian translated to English, judged by Gemini 3 Flash (not a participant in the benchmark), scoring 0–4 based on semantic equivalence rather than literal match.
• Cloze (50 blanks): Given 30 dictionary-form words (including 5 distractors), models must fill in the correct word inflected to the proper grammatical form.
• Error Correction (17 passages): Correcting planted grammatical, spelling, and punctuation errors without modifying originally correct content. Over-correction is penalized.

The scoring design for the error correction task is particularly elegant: it computes word-level edit distances before and after correction against the gold standard, using score = max(0, (errors_before - errors_after) / errors_before). Changing a correct word to an incorrect one also lowers the score — like a strict teacher.

Engineering Trade-offs Under Budget Constraints

The author's original plan was ambitious: nine dimensions, all 41 models in the Kaggle catalog, with an estimated full run costing around $345. But he then discovered that Kaggle's model credit system is $10 free per day, with no option to top up with your own money — there is simply no "add funds" button.

This single constraint reshaped the entire project: 41 models cut to 8, 9 dimensions cut to 5, and the dataset trimmed. The trimming itself reflected solid engineering thinking — a cheap "probe model" was run on the full dataset, retaining all questions it answered incorrectly and discarding some that were too easy, while large-number questions were kept as-is. The 186 number questions were reduced to 82, and the 115 verb questions to 50.

The actual total cost for the entire project came to $13.84. Notably, every time a task was pushed to Kaggle, it automatically ran on the default model Gemini 3.7 Flash and billed accordingly — these automatic runs accumulated nearly $3, more than half of what the final comparative runs cost. The lesson is clear: test locally, push less.

Models Tested and Rules for Fairness

Eight models from Anthropic and OpenAI were organized into four tiers:

Tier	Anthropic	OpenAI
Lightweight	Claude Haiku 4.5	GPT-5.4 mini
Mid-range	Claude Sonnet 4.6	GPT-5.4
Top-tier	Claude Opus 4.8	GPT-5.5
Latest ⚡	Claude Opus 5	GPT-6 Astra

To ensure fairness, all models had thinking disabled and answered directly — mixing thinking and non-thinking models would make comparison meaningless. However, the latest pair ran into a snag: GPT-6 Astra does not support disabling thinking at all, and could only be set to "low." Accordingly, Claude Opus 5 was also set to "low," allowing both to compete on equal footing, with both labeled ⚡ throughout the dashboard.

Additional rules included: if more than 5% of API calls failed, no score was reported (to avoid disguising "ran out of budget" as "model underperformed"); translation was consistently judged by Gemini 3 Flash; and partial credit was awarded for error correction.

Core Findings: Comprehension Is Solved, Generation Is Not

Here are the complete results:

Model	Numbers	Verbs	Translation	Cloze	Error Corr.	Average	Cost
Claude Haiku 4.5	0.30	0.34	0.96	0.54	0.25	0.48	$0.10
GPT-5.4 mini	0.41	0.42	1.00	0.52	0.67	0.60	$0.05
Claude Sonnet 4.6	0.39	0.52	1.00	0.76	0.44	0.62	$0.29
GPT-5.4	0.46	0.54	1.00	0.88	0.66	0.71	$0.15
Claude Opus 4.8	0.54	0.68	1.00	0.84	0.68	0.75	$0.58
GPT-5.5	0.51	0.60	0.98	0.88	0.74	0.74	$0.30
Claude Opus 5 ⚡	0.65	0.84	1.00	0.86	0.77	0.82	$1.22
GPT-6 Astra ⚡	0.83	0.90	0.98	0.96	0.82	0.90	$1.73

The most striking finding is the translation column: every model scored between 0.96 and 1.00. They can all translate Georgian into English near-perfectly. But the moment models are asked to generate correct Georgian — constructing numbers, conjugating verbs, filling in the right inflected form, correcting sentences — scores drop sharply. Without thinking enabled, even the strongest Claude Opus 4.8 answered nearly half of the number questions incorrectly.

Numbers Are the Final Boss

The numbers task was the hardest for 6 of the 8 models. The breakdown reveals striking disparities: reading Georgian numbers and converting to digits scored 0.74–0.96, but "words-to-digits for ordinals" ranged from 0.04 to 0.92, and large numbers (10¹⁵ to 10³⁹) scored only 0.06 to 0.53. The reason is that Georgian ordinals use either a prefix or suffix depending on the numeral (e.g., მე-18 vs. 21-ე), and most models defaulted to the more frequently seen pattern. Faced with მე-40000 ("the forty-thousandth"), only 2 of the 8 models got it right.

Georgian uses a vigesimal (base-20) number system — a minority choice among the world's languages, similar to French (80 = "four twenties") and Mayan numerals, but Georgian's implementation is more complex. For example, 97 must be expressed as ოთხმოცდაჩვიდმეტი, meaning "four×twenty plus seventeen," with dedicated composition rules at each level. The prefix/suffix distinction for ordinals (მე- for smaller ordinals, -ე for compound ones) is a context-sensitive rule that models easily overgeneralize from more common training patterns. For numbers above 10¹⁵, Georgian-language data is virtually nonexistent on the internet, meaning models are essentially doing zero-shot extrapolation — which explains the extremely low scores.

Errors That Are "Very Georgian"

The author's favorite part was reading the wrong answers — they weren't random noise, but rather looked like "speaking Georgian with someone else's grammar":

• "I fell in love with you" requires the inverted verb შემიყვარდი (literally "you became my beloved") — only GPT-6 Astra got it right; most others constructed a standard transitive verb.
• "We sat down" uses the plural-specific root ვისხედით; the author had planted the erroneous form ვიჯექით, and not a single model corrected it.
• "One million" is simply მილიონი, but most models wrote ერთი მილიონი ("one million" with an explicit numeral).

Cost and Tokens: Georgian Is More Expensive for Claude

Given identical prompts and identical Georgian text, Claude models consumed 2 to 2.6 times more input tokens than GPT models. For the numbers task, Claude averaged about 272 tokens per call versus 104 for GPT; for cloze, 1,412 versus 699. This is partly due to each provider's answer format instructions, and partly due to how their respective tokenizers segment Georgian script. The practical implication: for Claude users, the real cost of a Georgian-language prompt is significantly higher than the listed per-token price suggests.

Measured by "cost per correct answer," GPT-5.4 mini wins on value at $0.0003 per correct answer; GPT-6 Astra achieves the highest score but at roughly 21 times the unit cost.

Easter Egg: Gemini Shows Up Uninvited

The Gemini 3.7 Flash that automatically ran on every push — while not part of the official comparison — quietly scored around 0.89 on average, second only to GPT-6 Astra. The key observation: even with thinking set to "none," it still generated around 480 output tokens per number question (non-thinking models only produce about 16), as if it simply couldn't stop. This raises an interesting conclusion — for languages like Georgian, reasoning capability may matter more than model scale, as the two strongest models in the table both had "low" thinking enabled.

The root cause of the token consumption gap lies in tokenizer design. Major LLMs typically use byte pair encoding (BPE) or similar algorithms optimized for high-frequency languages — especially English. The Georgian alphabet (Mkhedruli), with its 33 characters, appears so infrequently in training data that tokenizers often cannot aggregate it into efficient multi-character tokens, falling back to byte- or character-level segmentation. This causes the same semantic content to occupy far more tokens than it would in English. Claude and GPT tokenizers handle Georgian script differently, which directly causes the 2–2.6× token gap. This phenomenon is not unique to Georgian — Arabic, Thai, Georgian, and other non-Latin-script languages broadly face "token inflation," and users should be mindful of this when estimating API costs.

Conclusions and Limitations

Final standings: OpenAI wins 3 of 4 tiers, with Anthropic's only win (top-tier, 0.75 vs. 0.74) essentially a draw. In the latest tier, GPT-6 Astra leads 0.90 to 0.82, taking the highest overall score with a particularly large margin on numbers.

The author is also candid about limitations: the dataset is "intentionally skewed hard," retaining mostly questions models got wrong, so absolute scores are low and rankings are the meaningful part; some tasks have only 12–17 questions, making small score differences statistically insignificant; each model was run only once, with approximately ±0.03 variance per task on repeat runs; and translation scores depend on the judge model's assessment.

This benchmark — completed by a single person under tight budget constraints — offers a methodological template worth borrowing for evaluating models on low-resource languages: hand-craft high-quality datasets, use a probe model for difficulty filtering, and apply rigorous engineering practices to ensure fair comparisons. It also surfaces an insight that extends well beyond Georgian: the current bottleneck for large language models isn't "reading comprehension" — it's "writing correctly."