From Word Vectors to Large Language Models: The Complete Evolution of AI 2.0 Technology

From AI 1.0 to AI 2.0: A Paradigm Shift

When we talk about ChatGPT and GPT-4, we're living in what's known as the "AI 2.0" era. So what distinguishes 1.0 from 2.0?

Large Language Models (LLMs) mark the boundary — everything before them belongs to the AI 1.0 era. That earlier phase was defined by various NLP engineering projects, characterized above all by poor generalization — models could typically only handle a single task, earning them the label "Narrow AI." The most iconic example is Deep Blue: it learned an enormous library of chess strategies and ultimately defeated the human world champion, yet it could only play chess and couldn't answer any other kind of question. Other representatives of that era include IBM's Watson (designed specifically for trivia competitions) and the early image classification network AlexNet — each reaching the pinnacle of its vertical domain while being unable to cross task boundaries.

So-called general intelligence

The true ultimate goal of AI has always been AGI (Artificial General Intelligence) — an agent with human-like intelligence capable of handling diverse problems across contexts. The concept of AGI was first articulated by philosopher John McCarthy and others at the 1956 Dartmouth Conference, and for decades it remained the "North Star" of AI research, existing largely as a thought experiment. This is precisely where large language models matter: they interact with us in human language, and through language as a medium, they can extend to an almost unlimited range of applications. The emergence of LLMs gave researchers the first tangible, quantifiable, and scalable engineering path toward AGI. It's this breakthrough in generalization that has convinced the field that following the path of large models might actually get us closer to AGI, step by step.

Where It All Began: Word Vector Technology

From a technical evolution perspective, the development of large models is a journey "from a point to a plane" — and that starting point is word vector technology.

The core idea behind word vectors is to represent words in natural language using mathematical vectors. For example, "cat" corresponds to a specific coordinate in vector space, "dog" to another, "cow" to yet another — each word has a unique coordinate, collectively forming a "word dictionary." The most common early representation was one-hot encoding: if the vocabulary size is 10,000, each word is represented as a sparse 10,000-dimensional vector with only a single dimension set to 1 and all others set to 0 — essentially labeling "apple" with a coordinate like 10.

The biggest advantage of this approach is that it converts the inherently ambiguous natural language into precise, computable mathematics, allowing machines to pinpoint every word exactly.

The efficiency problem

But word vectors have obvious shortcomings: First, they can't express relationships between words — the cosine similarity between any two one-hot vectors is always 0, so the model has no idea what connection exists between "cat" and "dog," let alone that "cat" and "dog" are more similar to each other than "cat" and "car." Second, they're representation-inefficient — the larger the vocabulary, the higher the dimensionality, and computational costs grow exponentially. These two problems became the direct drivers of subsequent technical progress, and directly gave rise to Word2Vec and word embedding technology.

Word Embeddings: Letting Vectors "Carry" Linguistic Information

To address the limitations of word vectors, the field developed word embedding (Embedding) technology. Interestingly, embeddings remain widely used in modern LLM projects today — they're a core concept you can't avoid when trying to understand modern AI.

The milestone in word embedding technology was Word2Vec, proposed in 2013 by a Google research team led by Tomas Mikolov. Its core idea comes from the "distributional hypothesis" in linguistics: a word's meaning is determined by the words that surround it. Through two training approaches — CBOW (predicting the center word from context) and Skip-gram (predicting context from the center word) — Word2Vec compresses words into dense vectors of 100–300 dimensions, giving rise to remarkable semantic arithmetic: "king − man + woman ≈ queen."

Word embeddings are essentially still word vectors, but with one critical addition: pre-training. By learning from vast amounts of text and documents, the originally "empty" word vectors become infused with rich linguistic information. Pre-trained word embeddings exhibit three important properties in vector space:

Low-dimensional representation: expressing words in a more compact form;
Semantic similarity maps to spatial proximity: words with similar meanings are located near each other in vector space;
Support for transfer learning: capabilities learned during training can be transferred to other tasks.

Task transfer

For example, by learning from large volumes of text, the model discovers that "cat" and "dog" are both pets, so their coordinates in vector space end up close together in a "pet" semantic region; meanwhile "cow" and "sheep" as common livestock cluster in another area. This "semantic clustering" capability was the earliest prototype of the pre-training idea behind large language models.

Sequence Modeling: From Sentence Vectors to RNNs and LSTMs

With word embeddings in place, models began moving toward "understanding context." This phase introduced sentence vectors and full-document vectors, with representative models being RNNs, LSTMs, and other recurrent neural networks.

Recurrent Neural Networks (RNNs) were inspired by the sequential nature of human reading: understanding a sentence requires remembering the words you've read before. RNNs pass information between time steps via "hidden states," and their core capability is processing sequential data — predicting what comes next given prior text, or translating one language into another. Take the sentence "What time is it?" — this type of model processes it word by word in order: first "What," then "time," then "is," "it," and finally the question mark, building understanding of the whole sentence through this sequential process.

In practice, however, RNNs suffer from a severe "vanishing gradient" problem — the further away information is from the current position, the more severely its gradient decays during backpropagation, meaning models can effectively only remember short-range context. In 1997, Sepp Hochreiter and Jürgen Schmidhuber proposed LSTM (Long Short-Term Memory), which introduced three learnable "gates" — forget gate, input gate, and output gate — allowing the model to selectively retain or discard historical information, dramatically extending the effective memory window and enabling "short-term memory" with "selective forgetting." Before the Transformer arrived, LSTM was the dominant architecture for machine translation and speech recognition, widely applied to text generation, speech recognition, image captioning, and more. By the RNN and LSTM era, many of the AI recognition capabilities we encounter in everyday life had already taken their early form.

The Pre-training Era: The Breakthrough of BERT and Transformers

Moving further, AI entered the phase of genuinely "understanding full-context" — the defining work being Google's BERT.

The most important breakthrough of this period was support for parallel training, powered by the Transformer architecture introduced by Google in their 2017 paper "Attention Is All You Need." Its core mechanism is self-attention: when processing each word in a sequence, the model simultaneously "sees" all other words in the sequence and computes attention weights to determine each word's contribution to the current position. This design breaks the sequential dependency of RNNs — all positions can be computed in parallel, and no matter how far apart two words are, the attention mechanism can directly establish a connection between them, fundamentally solving the long-range dependency problem.

Support for parallel training

BERT (Bidirectional Encoder Representations from Transformers) was released by Google in 2018, and its innovation lay in bidirectional pre-training: through the Masked Language Model (MLM) task, 15% of tokens in the input sequence are randomly masked, and the model must restore them using both left and right context, forcing it to learn truly bidirectional semantic understanding. BERT can handle "fill-in-the-blank"-style tasks: inferring from context whether a pronoun refers to a male, female, or animal subject, and completing the gap in a sentence. Upon release, BERT swept across 11 NLP benchmark tests, establishing "pre-training + fine-tuning" as the standard paradigm for NLP engineering and marking the formal beginning of the "true pre-trained model era."

The Era of Massive Models: Establishing the Prompt Paradigm

Finally, with OpenAI's ChatGPT as its hallmark, AI entered the era of massive models and unified multi-task capability.

The key to this transformation came from Google's T5 model, which introduced the Prompt paradigm for training models — feeding prompts into the model, training it to generate corresponding answers, and iterating repeatedly. When we ask ChatGPT a question, we're essentially guiding the model's output through prompts.

However, the original GPT-3, despite its 175 billion parameters, had an "alignment" problem — the model's outputs didn't always match human intent, sometimes generating harmful or nonsensical content. OpenAI addressed this with RLHF (Reinforcement Learning from Human Feedback): first collecting preference data from human annotators on model outputs, training a reward model, then fine-tuning GPT with reinforcement learning (PPO algorithm) to make its outputs better align with human expectations. This technical approach produced InstructGPT, and ChatGPT is essentially a conversational-optimized version of InstructGPT. This is OpenAI's true core contribution — systematically combining the "Prompt-based training paradigm" with RLHF alignment techniques and scaling it up broadly.

By the time we reach ChatGPT and GPT-4, model capabilities are remarkably impressive, and they've further extended into multimodal capabilities: not only understanding and generating text, but also generating images from text (text-to-image) and even generating video.

Summary: One "Point to Plane" Evolution

Looking back at the entire journey, the development of large language models can be clearly summarized as a single through-line:

Word Vectors → Word Embeddings (pre-training prototype) → Sentence Vectors / RNNs, LSTMs (sequence modeling) → BERT / Transformers (parallel training, full-context understanding) → ChatGPT / GPT-4 (massive models, Prompt paradigm, RLHF alignment, multimodality)

The most essential takeaway from this trajectory is: all modern AI capabilities are built on the single "point" of word vectors, evolving continuously through neural networks, from sequential to parallel training, and ultimately enhanced by RLHF alignment techniques to become today's unprecedented large language models. For developers hoping to enter the AI field, understanding this evolutionary history is the very first step toward seizing the opportunities of the AI 2.0 era.