#tokenization
123 related articles
·3 min
TokenTown: A Visual Approach to Understanding How LLMs Predict the Next Token
TokenTown is an open-source visualization project that intuitively presents the internal token prediction process of LLMs using a town metaphor. Learn its design philosophy and educational value.
Read more →
·3 min
Deep Analysis and Hardening Practices for 16 Critical Vulnerabilities in an AI Security Platform
An AI security platform was found to have 16 critical vulnerabilities spanning prompt injection, privilege escalation, and auth bypass. A deep dive into hardening methodologies.
Read more →
·2 min
Is Formal Language Theory Still Relevant in NLP? Deep Reflections Behind a Course Selection Dilemma
Formal Languages vs. Programming Language Principles—which course matters more for computational linguistics and NLP? A deep analysis from Chomsky Hierarchy to Lambda calculus to modern LLM theory.
Read more →
·2 min
Segue: A Deep Dive into the Cross-Platform AI Conversation Context Migration Tool
Segue is an AI context migration tool that uses short handles to seamlessly transfer conversation context across ChatGPT, Claude, and other AI platforms, solving the context-reset problem when switching tools.
Read more →
·2 min
Segue: A Deep Dive into the Cross-Platform AI Conversation Context Migration Tool
Segue is an AI context migration tool that uses short handles to seamlessly transfer conversation context across ChatGPT, Claude, and other AI platforms, solving the cross-platform context reset problem.
Read more →
·3 min
ChatGPT Now Handles Documents, Spreadsheets, and Slides: A New Breakthrough in AI Office Automation
ChatGPT adds document, spreadsheet, and slide processing capabilities, supporting information consolidation, data analysis, presentation generation, and in-file editing — directly competing with Microsoft Copilot and Google Workspace AI.
Read more →
·3 min
Replacing spaCy's Sentencizer with yasbd: Accuracy Jumps from 55% to 98.9%
spaCy's default Sentencizer achieves only 55.4% accuracy on edge cases, while open-source library yasbd reaches 98.9%. Analysis of limitations and integration code examples.
Read more →
·3 min
A Beginner's Guide to AI Agents: Two Learning Paths for Developers and Practitioners
How to learn AI Agents from scratch? This guide covers two clear paths: developers go from Python to LLMs to open-source framework source code; practitioners use Claude Code or similar tools to get results fast.
Read more →
·3 min
Claude Opus 5 Officially Released: Performance Approaching the Top Tier as the Open-Weight Debate Heats Up Again
Anthropic releases Claude Opus 5 with near-frontier performance at lower prices. Same day, Jensen Huang co-signs open-weight letter with 20+ companies while DeepSeek fundraising rumors surface.
Read more →
·3 min
Why Can't AI Draw Charts Properly? The Cognitive Blind Spots in LLM Visualization Capabilities
Top AI LLMs can write code and pass professional exams, yet can't produce an accurate chart. This article analyzes why AI fails at chart generation and reveals the uneven nature of AI intelligence.
Read more →
·3 min
AI Companies Destroying Rare Books for Training Data: The Cultural Cost Behind Efficiency
Some AI companies are buying rare antique books, using destructive scanning for training data, then destroying the originals. This raises urgent questions about AI data ethics and cultural heritage.
Read more →
·4 min
GenAI Job Interview Gone Wrong: Interviewer Googles Gemini for Questions on the Spot — What's Broken in AI Hiring?
A fresh grad interviewing for a GenAI Trainer role faced prime number coding and activation function questions while the interviewer used Gemini to generate questions live — exposing AI hiring chaos.
Read more →
·3 min
Wattage: An AI Agent Token Cost Profiling and Regression Protection Tool
Deep dive into Wattage, an AI Agent token consumption profiling and cost regression protection tool, exploring its core features, industry context, and value for developers.
Read more →
·3 min
Sub-Second Code Search: Building an Indexing Plugin with Cursor + Codex
How to achieve millisecond-level code search in massive repositories with hundreds of thousands of files? A hands-on look at building a code search plugin with Cursor + Codex, powered by Lucene inverted indexing and BM25 ranking.
Read more →
·3 min
Sub-Second Code Search: Building an Indexing Plugin with Cursor + Codex
How to achieve millisecond-level code search in a repo with hundreds of thousands of files? Building a search plugin with Cursor + Codex, powered by Lucene inverted indexing and BM25 ranking.
Read more →
Dive into LLMs: A Complete Guide to th…
·1 min
Dive into LLMs: A Complete Guide to the 44.8K-Star Hands-On LLM Tutorial
"Dive into LLMs" is a 44,830-star Chinese LLM tutorial on GitHub. Using Jupyter Notebooks, it covers Transformers, LoRA fine-tuning, RAG, and Prompt Engineering.
Read more →
·6 min
The AI Industrial Age Arrives: Compute Wars, Chip Reconstruction, and the GPT-5.6 Cheating Scandal Explained
Five key AI industry trends: Doubao surpasses 180 trillion daily calls, OpenAI's in-house AI chip, NVIDIA's $3-4 trillion compute forecast, China catching up, and the GPT-5.6 cheating scandal.
Read more →
Kronos Financial Foundation Model: Usi…
·7 min
Kronos Financial Foundation Model: Using AI to Read the Market Language of Candlesticks
Kronos is the first open-source foundation model treating candlestick data as the "language of financial markets," using an autoregressive Transformer and earning 32K GitHub Stars. A deep dive into its principles, applications, and limits.
Read more →
·5 min
Inside Cursor: How Recursive Model Improvement Enables AI to Self-Evolve
A Cursor ML engineer breaks down AI training methodology: outer/inner loop acceleration, preventing reward hacking, textual feedback, and recursive self-improvement (RSI) where models train the next generation.
Read more →
·6 min
Claude Sonnet 5 In-Depth Testing: 8 Real-World Tasks Reveal Its True Capability Boundaries
Bilibili creator KaterSony tests Claude Sonnet 5 across 8 real-world tasks—image recognition, 3D modeling, web generation—comparing it against GPT-5.5, Gemini 3.1 Pro, and revealing its true capability limits and cost traps.
Read more →