#subword tokenization
47 related articles
·3 min
SPA Tokenizer Fix Upgrade: Technical Analysis and Practice of a Wider Tokeniser
In-depth analysis of the SPA tokenizer fix and wider Tokeniser upgrade, exploring vocabulary expansion's impact on model performance, tokenizer mechanics, boundary handling fixes, and Playground verification.
Read more →
·2 min
10 Trillion Tokens in 3 Months: Open-Source Model Demand Is Exploding
Open-source LLMs processed 10 trillion tokens in under 3 months, hitting 300B daily. We break down what this milestone means and why open-source demand is accelerating.
Read more →
·2 min
Creating Characters with Fake Words: Unveiling a Midjourney Semantic Imagination Experiment
Explore how random fake words drive Midjourney to generate unique characters. Learn how AI interprets nonexistent vocabulary and how this method helps creators break prompt dependency and spark original character design inspiration.
Read more →
·3 min
Token Economics: The New Unit of Value Measurement in the AI Era
Deep dive into how Tokens evolved from a technical concept in LLMs to the core unit of measurement in the AI economy. Exploring Token consumption explosion, cost optimization, and Token economics.
Read more →
·4 min
SELENE Open-Source Project: An Interactive Notebook Library for Systematically Learning AI and Machine Learning from Scratch
SELENE is an open-source AI learning resource built on Jupyter Notebooks, systematically covering ML, deep learning, Transformers, and LLMs with interactive code and math derivations for beginners.
Read more →
·3 min
Dual RTX 3060 Running DeepSeek V4 Flash: IQ2_M Quantized Inference Benchmarked at 3.5 tok/s
Benchmarking DeepSeek V4 Flash on dual RTX 3060 GPUs with 96GB RAM at IQ2_M quantization achieving 3.5 tokens/sec. Covers hardware choices, 2-bit quantization techniques, and local LLM deployment optimization.
Read more →
·3 min
Why Can't Word2vec Handle Out-of-Vocabulary Words? OOV Problem Explained with Solutions
Deep dive into why Word2vec can't handle OOV words, explaining the lookup table mechanism and covering solutions like FastText subwords, UNK tokens, and BERT.
Read more →
·3 min
TokenTown: A Visual Approach to Understanding How LLMs Predict the Next Token
TokenTown is an open-source visualization project that intuitively presents the internal token prediction process of LLMs using a town metaphor. Learn its design philosophy and educational value.
Read more →
·3 min
Why Can't AI Draw Charts Properly? The Cognitive Blind Spots in LLM Visualization Capabilities
Top AI LLMs can write code and pass professional exams, yet can't produce an accurate chart. This article analyzes why AI fails at chart generation and reveals the uneven nature of AI intelligence.
Read more →
·6 min
The AI Industrial Age Arrives: Compute Wars, Chip Reconstruction, and the GPT-5.6 Cheating Scandal Explained
Five key AI industry trends: Doubao surpasses 180 trillion daily calls, OpenAI's in-house AI chip, NVIDIA's $3-4 trillion compute forecast, China catching up, and the GPT-5.6 cheating scandal.
Read more →
·4 min
Claude Usage Anxiety: The Hidden Cost Trap of AI Coding Tool Subscriptions
Real Reddit user rants reveal AI subscription pain points: Claude, Sol, and other tools consume usage at wildly different rates—does faster mean pricier? A deep dive into AI billing logic, usage transparency, and platform trust.
Read more →
·3 min
Render Code as Images and Send to Claude — Cut API Costs by 70%
PXPipe renders system prompts and tool docs as images sent to Claude, exploiting visual token pricing. Real-world results: 68% fewer input tokens, 59–70% lower bills.
Read more →
Cross-Tokenizer Knowledge Distillation…
·2 min
Cross-Tokenizer Knowledge Distillation: Why 85% of Information Vanishes and How the Chain Rule Fixes It
Many-to-one token mapping in cross-tokenizer knowledge distillation silently drops 85% of teacher info, collapsing entropy from 2.09 to 0.32 bits. Learn the chain rule fix that restores retention to 83%+.
Read more →
Running Gemma LLM in Godot with GDScri…
·2 min
Running Gemma LLM in Godot with GDScript and Vulkan: An Experimental Deep Dive
A developer runs the Gemma LLM inside Godot 4 using only GDScript and Vulkan compute shaders — no llama.cpp or external dependencies. A technical breakdown of how it works.
Read more →
·3 min
Training a 100M-Parameter Model for Under $50: A Complete Record of Building SmoLLM from Scratch
An ML engineer trained SmoLLM, a 109M-parameter LLaMA-style model from scratch for under $50. Full breakdown of architecture, training pitfalls, instruction tuning, and real-world performance.
Read more →
·4 min
From Enter to First Token: Decrypting the Full LLM Inference Pipeline (Autoregression, KV Cache & Decoding Strategies)
From pressing Enter to the first character appearing, what happens inside an LLM? This article breaks down autoregressive generation, KV cache acceleration, and decoding strategies like temperature, Top-k, and Top-p.
Read more →
AI Costs Out of Control: Real-World St…
·6 min
AI Costs Out of Control: Real-World Strategies for Enterprises to Rein In Soaring Bills
More enterprises are finding AI operational costs spiraling out of control. This article dissects token billing traps and blind flagship-model use, and maps out cost-reduction strategies like model routing, open-source self-hosting, and semantic caching.
Read more →
·5 min
How Can Working IT Professionals Learn AI Efficiently? A Practical Learning Path from Zero to Real-World Projects
Limited time but want to learn AI systematically? This guide maps out a practical learning path for working IT pros—from AI application engineering and prompt engineering to RAG and Agents.
Read more →
·5 min
Practical AI Cost Optimization: A Deep Dive into Intelligent Routing and Context Compaction
AI bills keep soaring? This article breaks down two core cost-reduction strategies: intelligent routing via an LLM Gateway, and context compaction to cut Token usage—no major refactoring needed.
Read more →
·5 min
GPT-5.6 Released: Hands-On Analysis of Programmatic Tool Calling and Autonomous Agents
GPT-5.6 is officially released with core upgrades including programmatic tool calling, autonomous subagent delegation, and higher token information density. A hands-on card game build reveals its Agentic power.
Read more →