Kaury Language Creator Tests: 36 AI Sessions Write Code, 17 Fail First Compile

Designer and language creator Kaury ran an interesting experiment—handing the spec of a small language that AI had never seen to 36 independent Claude sessions, asking them to write compilable code using only a spec document. The result isn't about how powerful AI is, but about how fragile language design can be.

Experiment Design: A Blind-Write Test Without a Compiler

Kaury can compile a single .kaury file into a complete website—pages, styles, logic, email-sending forms, 3D scenes, with server-side rendered HTML output, plus sitemap, structured data, and llms.txt. This language has one key constraint: no model has ever seen it in training, and AI assistants can only learn about it through a spec file.

The test rules were very clean: each session received a folder containing four photos, the spec document, and a request—a restaurant website, a quiz, a mortgage calculator, a data page with charts and sortable tables, etc. Sessions were asked to read the spec only, write in one shot, and not use a compiler to check or fix. Afterward, the author ran the code with the real compiler.

All participating sessions were brand-new Claude instances. The author explicitly stated that other models have not been tested, so the conclusions apply only to Claude.

The spec document as the sole source of knowledge was a deliberately designed constraint, aimed at excluding language patterns the model memorized from training data. In regular code generation tasks, large language models often rely on the vast amounts of Python, JavaScript, and other language samples in the training corpus, and can fill in the blanks by "linguistic feel" even with incomplete prompts. But when the target language is entirely absent from the training distribution, the model can only generalize and reason strictly based on the spec document in context. This kind of test is academically called an "out-of-distribution generalization" evaluation, and it more authentically reflects the model's ability to read technical documentation and act on it, rather than simply reproducing memorized patterns.

The rule of blind writing in one shot is equally critical. Prohibiting compiler checks and iterative revisions means sessions must complete the entire process from understanding the spec to producing code in a single generation, and any misreading or omission of language details is directly exposed as a compilation error. This "first-shot compile rate" metric reflects the informational completeness of the spec document better than scenarios that allow repeated debugging: if a document can enable multiple independent sessions to write runnable code without any feedback, the spec is clear enough; conversely, a large number of similar failure patterns points to ambiguities in the spec and the language design itself.

First Group: 11/20 Pass, and the Failure Reasons Are Almost All Reasonable

Of the first group of 20 files, 11 compiled on the first pass, and 9 failed. The author believes almost every failure was reasonable—the problem lay with the language or the spec itself, not the AI.

Extra commas after elements. The AI wrote row, gap 16, but the syntax requires row gap 16. The comma doesn't create ambiguity, and the parser has no good reason to reject it.

Listing properties below like CSS. The AI wrote:

column
  radius 12
  padding 24
  text "Hello"

But the language requires options to follow the element on the same line. This style is also unambiguous, yet it was rejected.

Non-existent French keywords. Kaury has French aliases, and when asked to write in French, sessions naturally guessed words like rayon for radius. But the spec document lists only four French words out of roughly two hundred.

Two fonts in reversed order. Every session put the heading font first, while the language expects the body font first. The site compiled, but the fonts were silently swapped—the author considers this the worst of the five failure categories, because "silent wrong results are more dangerous than errors."

Countdown displays as 0. Nothing in the language can convert text to a date.

Silent error refers to a program that compiles or runs normally but produces results inconsistent with developer intent, without throwing any warning or exception. In the Kaury font-order case, the compiler swapped heading and body fonts, and the page still rendered—just visually off from expectations. This type of problem is recognized in software engineering as one of the most dangerous categories of defects, because it bypasses all automated checking mechanisms: unit tests that only verify "compilation succeeded" will all pass, and code reviews that only look at structure will miss semantic deviations. In contrast, a compilation error at least provides a clear failure signal and location information, which developers can immediately use to fix.

In language and API design, silent errors typically stem from two situations: first, multiple parameters share the same type with no naming constraint, so order errors aren't caught by the type system; second, defaults mask missing configuration, so misconfiguration is silently "backstopped." Kaury's font declarations happen to be the first case—heading and body fonts are both string parameters, and the compiler can't distinguish them by type. Common fixes for this type of problem include requiring named parameters, giving examples at the declaration site, or introducing configuration objects to replace positional parameters.

The Author's Choice: Change the Language, Not Write Better Prompts

Faced with these failures, the author didn't write more detailed instructions, but directly modified the language so that reasonable guesses become correct usage:

• Commas after elements are now accepted;
• Options listed below elements are accepted;
• French words are generated from the compiler's own vocabulary, with tests ensuring the spec document is accurate;
• Font order is shown by example directly at the font declaration site;
• Added to-date and format-number.

Re-testing with the same 20 files, 18 compiled directly. After another round of iteration letting sessions receive compiler error messages, 19 were achieved.

Those error messages are themselves written for this purpose—giving line numbers, positions, and suggestions:

Error site.kaury, line 15: "likez" does not exist.
  15 |     button "Go" -> likez += 1
     |                    ^^^^^
Try: declare it first: let likez = …   (or state likez = … if it changes)

The design of compiler error messages occupies a central position in developer experience. A high-quality error message typically contains three elements: precise location (filename and line number, even column range), a clear diagnosis (pointing out where the error is and what was expected), and actionable suggestions (what to do next). Kaury's example satisfies all three at once: the line number line 15 locates the problem, "likez" does not exist explains the cause, the arrow ^^^^^ points to the specific symbol, and finally Try: guides the fix.

Such messages are especially effective for AI sessions because the model can directly read and adopt the suggestions in the next iteration, essentially turning the compiler into a feedback loop. Traditionally error messages are written for human developers, but in AI-assisted programming scenarios, they also become an interface for human-machine collaboration. This also explains why the author, in the third round of iteration, raised the pass rate from 18/20 to 19/20 merely by letting sessions receive error messages—the error message itself is a highly compressed, structured prompt.

The Last Two Groups: 16/16 All Pass

Two new groups of eight sessions took on harder requests, and all 16 files compiled with no errors or warnings. This shows that after targeted fixes, the success rate for AI writing Kaury code from the spec document alone improved significantly—but the author cautions that this test still covered only one model, Claude.

The limitation of sample size on extrapolating conclusions deserves special attention. The first group had 20 sessions, and the latter two groups had 8 sessions each, totaling only 36 independent instances, all from the same model family. At this scale, while the improvement in pass rate from 11/20 to 18/20 is significant, the confidence interval is wide; the 16/16 all-pass result may not reliably reproduce on larger samples or more complex tasks. The author himself repeatedly emphasizes in the article that "other models have not been tested" and "conclusions apply only to Claude"—this self-limitation is necessary caution in experimental research.

Moreover, there's a clear gap between "compiles" and "recommends using": 13/20 in the first group would actively choose Kaury, and in each of the two harder groups only 2/8 would. This shows that technical success and product adoption are two independent dimensions. The reasons given by sessions—can't self-check, it's just a small component, there are more specialized generators, needs payment features—point to scenario-fit issues, not language capability issues. For evaluating any new tool or new language, these two types of metrics should be measured separately; otherwise it's easy to overestimate actual value.

The Question the Test Can't Answer: Can Write It Doesn't Mean Will Choose It

The author also asked each session: if deciding for yourself, would you recommend Kaury for this request? In the first group of 20, 13 said yes; in the two harder groups, only 2 out of 8 said yes each. The reasons given were reasonable: no compiler on hand to self-check, the request is really just a small component of an existing site, a docs site would be better with a specialized generator, needs payment features.

Another key limitation: compiling doesn't equal looking good. The font reversal is evidence. The author had to personally look at the pages to judge the effect, and that doesn't scale.

Kaury itself isn't omnipotent either; the spec document clearly states: no account, database, or payment support, and it can't be embedded into an existing site as a component.

The Real Insight of the Experiment

The value of this experiment isn't in proving AI can write code, but in demonstrating a reusable method for testing language design: using AI's "reasonable guesses" as probes to find unnecessary strictness and ambiguity in a language.

When a new language's spec document is the AI's only source of knowledge, the accuracy of the spec and the language's tolerance become equally important. The author chose to accept common incorrect usages rather than require the AI to change habits—this trade-off is worth thinking about for anyone designing new tools, new APIs, or new products.

The language's official website is itself written in Kaury, showcasing six running sites and their code, with a desktop Lighthouse^Ad score of 100, most mobile pages at 98 to 100, and the homepage at 93 due to running a 3D scene. The install command is:

npm install -g @kaury/cli