#Tokenizer
201 related articles
·3 min
SPA Tokenizer Fix Upgrade: Technical Analysis and Practice of a Wider Tokeniser
In-depth analysis of the SPA tokenizer fix and wider Tokeniser upgrade, exploring vocabulary expansion's impact on model performance, tokenizer mechanics, boundary handling fixes, and Playground verification.
Read more →
Cross-Tokenizer Knowledge Distillation…
·2 min
Cross-Tokenizer Knowledge Distillation: Why 85% of Information Vanishes and How the Chain Rule Fixes It
Many-to-one token mapping in cross-tokenizer knowledge distillation silently drops 85% of teacher info, collapsing entropy from 2.09 to 0.32 bits. Learn the chain rule fix that restores retention to 83%+.
Read more →
·4 min
Muse Glimmer Local Testing: Meta's Open-Source 30B Multimodal Model Runs on a Single GPU
Meta releases Muse Glimmer, a 30B open-source multimodal model running on a single 24GB GPU. Tested at 233 tokens/sec with speculative decoding on RTX 5090, Apache 2.0 licensed with GGUF support.
Read more →
·3 min
Deep Analysis of Cursor Pay-Per-Use Plugins: The Truth and Risks Behind a 75%-Off "Legit" Solution
Deep analysis of Cursor pay-per-use refill plugins: account pool rotation mechanics, technical logic, and three key risks including compliance, data security, and prepaid fund loss.
Read more →
·2 min
Gemma 4:31b Hands-On: Major Stability Improvements in Ollama Local Deployment
Reddit user reports Gemma 4:31b on Ollama is now much more reliable: tool calls no longer fail frequently and gibberish output issues are gone.
Read more →
·2 min
NVIDIA Open-Sources Switchyard: An AI Task Scheduling Engine Built with Rust
NVIDIA-NeMo team open-sources Switchyard, a high-performance AI task scheduling engine built in Rust. Explore its technical positioning, why Rust was chosen, and its strategic role in the NeMo ecosystem.
Read more →
·3 min
Is Claude Secretly Watermarking AI Content with Steganography? A Deep Dive into the Closed-Source Model Trust Crisis
Reddit users accuse Claude of using steganography to secretly mark AI content, sparking a closed-source transparency debate. We analyze the tech, false positive risks, and open vs closed model trust.
Read more →
·3 min
Why Does Gemini Keep Getting Things Wrong? A Deep Dive into AI Hallucinations and How to Deal with Them
Deep analysis of why Google Gemini and other LLMs frequently produce errors, explaining the technical mechanisms behind AI hallucinations and offering practical prompting tips for better AI usage.
Read more →
·3 min
Prompt Caching Explained: The Key Technology for Saving Money with AI Agents
Deep dive into Prompt Caching: how it works, why AI Agents repeatedly send tokens causing costs to skyrocket, and best practices to slash LLM costs by up to 90%.
Read more →
·2 min
A Go Developer's Guide to Gemini: Complete Model Family Breakdown
A comprehensive Gemini model family guide for Go developers, covering Pro vs Flash selection strategies, multimodal capabilities, official Go SDK integration, and token management practices.
Read more →
·4 min
Edit Distance Automata and N-gram Similarity: Core Technologies Behind Fuzzy Search in Search Engines
Deep dive into two core fuzzy search technologies: Levenshtein edit distance automata for efficient spelling tolerance and N-gram similarity for large-scale candidate recall, with principles, performance comparison, and industrial integration.
Read more →
·3 min
Latent Space Reasoning: How DeepSeek-V4 Lets AI Think in Hidden Layers
Deep dive into DeepSeek-V4's latent space reasoning technology — how AI shifts from explicit chain-of-thought to implicit vector space reasoning, its efficiency gains, and challenges in interpretability.
Read more →
·2 min
Google SDK Code Leak Reveals Gemini 4 Flash: A New Lightweight Model on the Horizon?
Google SDK tokenizer code accidentally reveals a gemini-4-flash-preview identifier, suggesting a new Gemini 4 Flash model is in development. We analyze the leak details, Flash series positioning, and naming possibilities.
Read more →
·2 min
Qwen3 Priced at Just $2/Million Input Tokens — The LLM Price War Intensifies
Alibaba's Qwen3 model priced at $2/million input tokens and $6 output, far below mainstream closed-source LLMs. Analysis of pricing logic, comparison with Claude, and the open vs closed-source debate.
Read more →
·3 min
Qwen3-Max Deep Dive: How Coding and Collaboration Capabilities Are Redefining AI Development Assistants
Deep analysis of Alibaba's flagship model Qwen3-Max, covering its coding, Cowork collaboration capabilities, and potential for redefining AI-assisted software development.
Read more →
·3 min
OpenAI Prepaid Credits Vanish Without Records: An Analysis of API Billing Transparency Issues
A developer found OpenAI prepaid credits marked consumed with no usage records available. We analyze API billing transparency issues and offer practical self-protection tips.
Read more →
·2 min
Alibaba's Qwen Surges to #2 on Text Arena — A Deep Dive into the Model's Capabilities
Alibaba's Qwen LLM surges to #2 on Text Arena via blind human evaluation, showcasing top-tier alignment quality. Analysis of Qwen's technical strengths, open-source strategy, and industry impact.
Read more →
·3 min
Building AI from Scratch: A Complete Learning Path with 503 Lessons in This Open-Source Course
AI Engineering from Scratch is an open-source course with 503 lessons across 20 phases, from linear algebra to autonomous agents, emphasizing hand-implementation before frameworks, supporting Python/TypeScript/Rust/Julia, with 46K+ GitHub stars.
Read more →
·2 min
Getting Started with Local AI Model Deployment: A Complete Guide from Downloading on Hugging Face to Running with llama.cpp
A beginner-friendly guide to local AI model deployment, covering secure model downloads from Hugging Face, running inference, exporting to GGUF format, and high-performance local execution with llama.cpp.
Read more →
·2 min
Creating Characters with Fake Words: Unveiling a Midjourney Semantic Imagination Experiment
Explore how random fake words drive Midjourney to generate unique characters. Learn how AI interprets nonexistent vocabulary and how this method helps creators break prompt dependency and spark original character design inspiration.
Read more →