Why Large Language Models Are Starting to 'Invent Words': The Nature of LLM Language Boundaries

A Thought-Provoking Community Observation

Recently on Hacker News, developers noted that top-tier large language models (LLMs) with strong narrative and creative capabilities appear to be constrained by "normal vernacular" — they seem unable to precisely express their internal representations using existing language.

This observation touches on a real phenomenon in the evolution of AI language models: when a model's semantic understanding surpasses the coverage of human vocabulary, its generated content begins to "invent words," drift toward obscurity, or produce expressive distortion.

The "top creative models" referenced here aren't an official vendor category, but rather an informal community term for cutting-edge LLMs with strong narrative, abstract expression, and imaginative capabilities. Understanding why they seem "constrained by ordinary vocabulary" is a key to understanding the language boundary problem in large models.

What Does "Being Constrained by Vocabulary" Mean?

Language as a Natural Boundary of Model Capability

At their core, large language models are probability distribution models trained on massive amounts of human text. In theory, everything they output originates from existing human linguistic data. The paradox, however, is that the semantic space (latent space) formed inside the model may be far more nuanced, continuous, and high-dimensional than the human vocabulary system.

What is a latent semantic space? In neural network architectures, every word, sentence, and concept is mapped to a point or region in a high-dimensional vector space. GPT-4, for instance, may have internal representations spanning tens of thousands of dimensions, each capturing some abstract semantic attribute. This representational approach traces back to early word vector research like Word2Vec and GloVe — where researchers discovered that vector arithmetic could capture semantic relationships (e.g., "king − man + woman ≈ queen"). Modern Transformer models push this idea to the extreme: through Self-Attention mechanisms, the model dynamically builds weighted associations between every token and all other tokens in a sequence, encoding extremely fine-grained semantic distinctions in continuous high-dimensional space — including context dependence, emotional tone, cultural meaning, and other complex attributes. Notably, this high-dimensional representation doesn't have each dimension independently carrying a single meaning. Recent Mechanistic Interpretability research from MIT and Anthropic has found that models contain large numbers of "polysemantic features" — individual neurons may respond simultaneously to multiple seemingly unrelated semantic concepts — making the true structure of the representation space far more complex than initially imagined, and much harder to fully map onto natural language.

In other words, some semantic nuances that models "perceive" in their vector space have no corresponding vocabulary in natural language. When models attempt to output these nuances, they can only approximate a more precise internal representation using rough, conventional words — this is the fundamental source of the "constraint."

From "Inventing Words" to "Expressive Distortion"

This tension typically manifests in three ways in practice:

• Tendency to coin terms: The model generates neologisms or unusual word combinations to fill expressive gaps
• Style drift: Output becomes highly abstract, poetic, or even obscure, approaching hard-to-express semantics
• Rhetorical excess: Long-form creative writing exhibits a style that "transcends the conventional"

From an engineering perspective, this is both a sign of capability and a signal of declining controllability. Once a model's expression moves beyond familiar vocabulary, the readability and reliability of its output both diminish.

Why Does This Tension Exist?

High-Dimensional Semantic Space vs. Finite Linguistic Symbols

Modern large models' embedding spaces typically have thousands of dimensions, capable of encoding extremely rich semantic relationships. But human natural language vocabulary, however vast, is ultimately a discrete and finite symbolic system. When a model attempts to "compress and project" its high-dimensional, continuous internal representations onto finite vocabulary, information loss is inevitable.

From an information theory perspective, this process is essentially Lossy Encoding. Information theory pioneer Claude Shannon observed that every channel has a capacity limit; when the entropy of the source (model internal representations) exceeds the channel capacity (the expressive space of the vocabulary system), information loss is unavoidable. Take English as an example: the Oxford Dictionary contains roughly 170,000 words, shaped by human cognitive needs — not designed to exhaust all possible semantic distinctions. The representational complexity of modern large models has already surpassed this "channel capacity," meaning the mapping from representation to output inevitably involves semantic compression and distortion. Coining new words is a spontaneous coping strategy by which models attempt to "expand channel capacity."

The information theory background of lossy encoding: Shannon's 1948 paper A Mathematical Theory of Communication laid the foundation for information theory. One of its core insights is that the efficiency of information transmission is limited by Channel Capacity — information beyond that capacity will inevitably distort in some form. In multimedia, JPEG image compression and MP3 audio compression are classic examples of lossy encoding — they trade storage efficiency for detail by discarding information below human perceptual thresholds. Language as a channel fits the same framework: linguists Chomsky and Pinker have both noted that human language is not a complete system exhausting all concepts, but rather a "lossy compression format" shaped by evolution and culture. When an AI model's representational complexity exceeds this format's carrying capacity, "semantic distortion" becomes a structural inevitability rather than an occasional error. This also theoretically explains why simply expanding vocabulary cannot fundamentally solve the problem — the real bottleneck lies in the architectural limitations of discrete symbolic systems themselves.

This is like trying to reproduce a painting with infinite color gradients using a limited palette — some subtle shades simply cannot be precisely rendered. Being "constrained by vocabulary" is essentially the visible manifestation of this projection loss at the generation layer.

The Inherent Conflict Between Training Objectives and Creativity

Mainstream LLMs are trained to predict the next token, which naturally biases them toward generating the "most likely" conventional expressions. But in creative tasks, we expect models to break conventions and produce novel content. There is a fundamental tension between these two goals.

The underlying logic of token prediction: A token is the basic unit models use to process text — it may be a word, a morpheme, or even a punctuation mark. GPT-series models typically use BPE (Byte Pair Encoding) tokenization, with roughly 1.3 words per token in English. During training, the objective is to maximize the probability of correctly predicting the next token given the context, iteratively optimized over trillions of tokens of data. This mechanism naturally biases the model toward "statistically most likely" expressions, because high-frequency, conventional word combinations appear most often in the training corpus and produce the strongest loss function gradients. When a model is encouraged to explore low-probability expression spaces, it is effectively working against the dominant tendency of its own training objective. Researchers have tried to address this conflict in various ways: introducing "creative writing fine-tuning datasets" in post-pretraining stages, using Contrastive Decoding to deliberately suppress overly conventional output paths, and rewarding "novel but coherent" outputs through reinforcement learning. However, all these methods face a common challenge: how to objectively quantify "creativity" itself remains an open, unresolved problem.

• Adhering too closely to the probability distribution → mediocre, uncreative output
• Deviating too far from the mainstream distribution → distorted, hard-to-understand output

What makes cutting-edge creative LLMs distinctive is precisely that they explore more aggressively between these two extremes — and thus more frequently brush up against the boundaries of linguistic expression and produce neologisms.

Practical Implications for AI Applications

Creative Scenarios: The Boundary as an Advantage

For literary creation, poetry generation, brand naming, and similar scenarios, the model's ability to "break through conventional vocabulary" is actually a competitive advantage. It can produce expressions humans haven't thought of, serving as an "inspiration trigger" that breaks cognitive patterns. Many designers and writers are already actively leveraging unconventional expressions generated by LLMs to spark creativity.

Controllability and Alignment: Active Constraints Required

In scenarios requiring precise and reliable output, this tendency must be actively managed. Prompt engineering, temperature parameter tuning, and output constraint mechanisms are the most commonly used practical tools for addressing this issue.

The technical principles behind temperature and prompt engineering: The temperature parameter directly affects the softmax probability distribution during model generation — as temperature approaches zero, the model almost always selects the highest-probability token, producing highly deterministic and conservative output; as temperature rises, the distribution flattens and low-probability tokens get more "airtime," making output more diverse and even unpredictable. Generally, a temperature of 0.7–1.2 is recommended for creative writing, while below 0.3 is recommended for precision tasks like code generation and factual Q&A. Beyond temperature, Top-P (nucleus sampling) and Top-K sampling are also common parameters — Top-P samples from the smallest set of tokens whose cumulative probability exceeds threshold P, while Top-K selects only from the K highest-probability candidates; both constrain the "adventurousness" of generation from different dimensions. Prompt engineering intervenes from another angle: using System Prompts to explicitly specify output format, style constraints, and vocabulary range effectively pre-establishes a "language interface specification" before the model generates, guiding it to map internal representations onto a more controllable vocabulary subspace.

How to keep models creative without losing control is also an important topic in current alignment research. It's worth noting that mainstream alignment methods — including OpenAI's RLHF (Reinforcement Learning from Human Feedback) and Anthropic's Constitutional AI — implicitly constrain output language style: human annotators naturally prefer expressions that conform to conventional language habits, and this preference is back-propagated through reward signals, causing the model's language style to converge toward "human-comprehensible" expression. However, this raises a deeper alignment paradox: if a model's true "understanding" is more refined than the language it can output, can alignment evaluations based on language output truly capture the degree of alignment in the model's internal states?

Frontiers in interpretability research and the alignment paradox: This paradox is one of the core motivations behind the rise of Mechanistic Interpretability research. Anthropic's research team uses techniques like Activation Patching and Circuit Analysis to directly interpret the model's internal computational processes, rather than inferring the model's "intent" solely from output text. The "Sparse Autoencoder" research published in 2023 further demonstrated that models contain millions of identifiable "conceptual features," many of which correspond to fine-grained semantic distinctions for which human language has no precise vocabulary. These findings suggest that alignment evaluations based solely on language output may systematically underestimate or mischaracterize the model's true internal state. How to build reliable mappings between a model's "internal language" and the "external language" comprehensible to humans has become a key challenge for AI safety moving from theory to engineering practice — this question currently has no definitive answer and remains a frontier research topic.

Language Evolution: The Reverse Influence Cannot Be Ignored

From a longer-term perspective, if people use these models extensively for creation and communication, the models' preferred expressions — and even the words they "invent" — could feed back and influence the natural evolution of human language.

Language is not a static system but a living structure that continuously evolves with social and cultural change — this is a basic tenet of sociolinguistics. Throughout history, every major media technology (the printing press, the telegraph, the internet) has profoundly reshaped human language habits.

The historical arc of media technology and language evolution: The 15th-century spread of Gutenberg's printing press drove language standardization, gradually converging regional dialects toward a unified written norm; the 19th-century telegraph gave rise to "telegraphese" — a highly compressed, economical style developed to minimize transmission costs, whose influence persists in modern journalistic writing conventions; the internet and social media incubated entirely new linguistic units like emoji and internet abbreviations (LOL, OMG) within decades, some of which have been formally incorporated into the Oxford English Dictionary. Linguists call this phenomenon "Technology-Driven Language Change." The intervention of AI large models represents a new phase of this process, distinctive in its scale and speed: hundreds of millions of users worldwide interact daily with the same set of models, and the linguistic preferences embedded in model outputs enter global language circulation at an unprecedented pace. Researchers at Carnegie Mellon University and Stanford University have begun tracking the diffusion of "AI writing signature vocabulary" in academic papers and social media; preliminary results show that the usage frequency of certain model-preferred words (such as "delve," "intricate," and "tapestry") has shown statistically significant increases since 2023.

The large-scale penetration of AI large models is opening a new phase of acceleration: large numbers of users directly copy or rewrite model outputs, introducing model-preferred expressions into everyday circulation; words "invented" by models that gain community recognition can rapidly spread and solidify via the internet; moreover, non-English language communities using models trained predominantly on English data face the risk of their native linguistic expression habits being quietly eroded by English-centric thinking patterns. Linguists have begun systematically tracking this phenomenon, but empirical research is still in its early stages given the longer time scales of language change.

This is a sociolinguistic phenomenon worth continued attention, whose impact may prove deeper than we anticipate.

Conclusion: Rational Observation

It must be noted that describing models as "feeling constrained" is an anthropomorphic expression — we should not interpret it as models having genuine "intent" or "subjective experience." A more accurate description is: the internal representational capabilities of models are exposing the expressive limitations of human natural language symbol systems.

This phenomenon reminds us that as large model capabilities continue to improve, the "language interface" between humans and machines is itself becoming a key area worthy of deep research. Understanding the gap between what models "want to express" and what they "can express" may be an unavoidable step toward more powerful and more controllable AI systems.

Key Takeaways