[KongchangAI]
· 2 min read· 1,121 words

AI-Written Model Runs on DOS: 367K Parameters Fit on a Floppy Disk

AI-Written Model Runs on DOS: 367K Parameters Fit on a Floppy Disk

AI-written, AI-trained 367K-parameter model compressed to 435 KB runs on DOS floppy-disk-era hardware.

An experimental micro language model autonomously written and trained by AI drew discussion: Claude wrote the Transformer, Opus 5.5 generated ~3,500 training samples, and the final model has only 367K parameters. After 8-bit integer quantization, program plus weights total ~435 KB, fitting on a 3.5-inch floppy and running in DOS memory on an Intel Pentium. It answers seen questions decently but outputs gibberish on complex or long ones. Its value lies not in performance but in validating the 'AI building AI' loop and the feasibility limits of language models under extreme constraints.

When Language Models Return to the Floppy Disk Era

A tiny language model written and trained by AI itself has recently sparked discussion on Reddit. The project had Claude (specifically version Fable 5.2) write the Transformer architecture, then had Opus 5.5 generate training data — roughly 3,500 facts, small talk, and math Q&A, each asked and answered in different ways. The final model has only 367K parameters, runs on an Intel Pentium processor, and the program plus weights total about 435 KB — enough to fit on a single floppy disk and load entirely into DOS memory.

This isn't a performance-oriented engineering effort, but a boundary experiment about the "minimal viable language model."

reddit source: Claude wrote and trained a language model for DOS that fits on a floppy disk

Technical Breakdown: How to Squeeze a Model onto a Floppy Disk

Data and Training

The training data was generated by Opus 5.5, covering about 3,500 factual knowledge items, simple chitchat, and arithmetic problems, with each item expressed in multiple phrasings to strengthen the model's generalization. The entire training process was completed by AI, with humans only setting goals and verifying results.

The Transformer architecture is a neural network structure proposed in the 2017 paper "Attention Is All You Need," whose core is the self-attention mechanism: every position in a sequence can directly "attend" to all other positions, thereby modeling long-range dependencies, unlike RNNs that pass state step by step. A complete Transformer typically includes embedding layers, multi-head attention, feed-forward networks, residual connections, and layer normalization. What's special about this project is that the architecture code wasn't hand-written by humans but directly generated by a language model, so extreme simplifications had to be made to parameter scale, layer count, and vocabulary size to fit within the 367K-parameter and DOS-memory constraints.

Quantization and Storage

Model weights are stored as 8-bit integers, with one scale factor per row. This quantization approach drastically compresses size, keeping the program and weights together at about 435 KB. For reference, a classic 3.5-inch floppy disk holds 1.44 MB — the model occupies only a fraction of it.

8-bit integer quantization means mapping the trained 32-bit floating-point weights onto the integer range of -128 to 127, compressing each parameter's storage requirement to a quarter of the original. The mapping requires a scale factor: if an entire row of weights shares one scale (per-tensor/per-row), the whole row shares the same dynamic range, causing greater precision loss but minimal metadata overhead; if each weight gets its own scale (per-channel), precision is higher but the extra storage eats into the compression gains. Scale per row is a compromise between the two — each row of matrix elements shares one scaling coefficient, controlling precision loss while adding only a few extra bytes. This idea is also the precursor to modern quantization formats like GGUF and GPTQ, and is especially common in embedded and edge inference.

Runtime Environment

The model runs on an Intel Pentium processor, executing entirely in DOS memory, processing letter by letter when reading and writing. This means it doesn't rely on modern GPUs or large memory, but returns to the computing constraints of early personal computers.

The DOS memory model differs greatly from modern operating systems: classic DOS real-mode addressing tops out at 1 MB, of which 640 KB is conventional memory, and applications must load code, data, and stack within this range. This means model weights can't simply be "mapped" into memory but must be carefully loaded and overwritten in segments. Additionally, letter-by-letter (character-level) processing means the model uses individual characters as the smallest token unit, rather than subword units like BPE or WordPiece; a character-level vocabulary is smaller and saves parameters, but sequences are longer and generation is slower, which also explains why long text output becomes incoherent.

Capability Boundaries: Good on What It's Seen, Nonsense on What It Hasn't

The author gave a candid assessment of the model's performance: for vocabulary and facts seen during training, it answers "quite well"; but when encountering complex or overly long questions, the output becomes "complete gibberish."

This reveals the fundamental limitation of tiny models — 367K parameters can hold an extremely limited amount of knowledge. It's more like a highly compressed question-answering lookup table than a general model with reasoning ability. The letter-by-letter read/write approach also means generation speed is limited by character-level processing, unable to smoothly output long text.

Why This Matters

The AI-Building-AI Loop

The most interesting part of the project isn't the model itself, but the development workflow: one AI writes the Transformer code, another AI generates the training data, and humans only orchestrate and evaluate. This "AI building AI" cycle is becoming a new engineering paradigm, even though the current results are extremely small in scale.

Engineering Thinking Under Extreme Constraints

In environments with severely limited compute and storage, 8-bit integer quantization and per-row scaling are pragmatic choices. Such techniques also have reference value in edge devices and embedded scenarios — when a model must fit into limited memory, how to trade off precision versus size is the core question.

A Counter-Reminder to the "Big Model Race"

As the industry chases trillions of parameters, this 435 KB model reminds people how low the capability floor of a language model can go, and how deep the chasm is between parameter scale and practical capability. It isn't practical, but it clearly marks the distance between "it runs" and "it works well."

Limitations and Outlook

This project is currently a technical demonstration with no practical application value. Its significance lies in verifying the feasibility of an extreme path: on antique platforms like DOS and floppy disks, language models can still exist. In the future, combined with more efficient quantization methods and larger training data, similar ideas might find a place in retro computing or educational demonstration scenarios.

For developers interested in AI programming and edge deployment, this experiment provides a reproducible minimal sample — 367K parameters, 435 KB, one floppy disk, enough to illustrate many things.

One-line Summary

An AI-written and AI-trained 367K-parameter language model compressed to 435 KB runs on DOS floppy-disk-era hardware.

Paragraph Summary

An experimental micro language model autonomously written and trained by AI has sparked discussion: Claude wrote the Transformer architecture, Opus 5.5 generated about 3,500 training samples, and the final model has only 367K parameters. After 8-bit integer quantization, the program and weights total about 435 KB, fitting on a single 3.5-inch floppy disk and running in DOS memory on an Intel Pentium. It answers questions seen during training reasonably well but outputs chaos on complex or overly long questions. The project's value isn't in performance but in validating the "AI building AI" development loop and the feasibility boundaries of language models under extreme compute and storage constraints.

#tiny language model#8-bit quantization#DOS#floppy disk#AI building AI#edge deployment
Share:

Related articles