Running a 90M-Parameter LLM on a Sony PSP: Where Are the Limits of Edge Inference?

When a Retro Handheld Meets a Modern Language Model

The Sony PSP (PlayStation Portable), released in 2004, came equipped with a 333MHz MIPS R4000 processor and 32MB of RAM — flagship portable gaming hardware for its era. The MIPS R4000 is a 64-bit microprocessor architecture introduced by MIPS Technologies in 1991, based on a Reduced Instruction Set Computing (RISC) design. Its core philosophy is to improve per-instruction execution efficiency by reducing instruction set complexity — a stark contrast to Intel's x86 Complex Instruction Set Computing (CISC) approach. The PSP actually uses the Allegrex processor, a custom design based on the MIPS R4000 architecture, with a clock speed dynamically adjustable between 1MHz and 333MHz. MIPS architecture was widely used in embedded devices, routers, and game consoles throughout the 2000s, but its lack of vector processing units and floating-point parallel processing capabilities puts it at a natural disadvantage for matrix-intensive computation like neural network inference.

Yet recently, a developer successfully ran a 90M-parameter conversational large language model on this "antique" device and open-sourced the project on GitHub (LLMPSP). This isn't just a tech geek experiment — it's a real-world measurement of the limits of edge inference.

What makes this noteworthy isn't its practicality, but rather the core question it reveals: On extremely resource-constrained hardware, where exactly are the boundaries of language model inference?

The PSP's Hardware Limits: 90M Parameters Is the Ceiling

Hardware Specs and Inference Bottlenecks

The PSP's computing power lags behind modern inference hardware by several orders of magnitude. A single-core CPU at 333MHz, no GPU acceleration, and 32MB of main memory — by today's standards, these specs wouldn't even qualify as entry-level embedded hardware. After testing, the developer concluded that a 90M-parameter model is the upper limit the PSP can handle at an acceptable inference speed.

Measured inference speed was approximately 0.5 to 0.6 tokens per second, with generating a complete response taking 1 to 3 minutes. It's worth explaining the concept of a "token" here: a token is the basic unit of text that a large language model processes. In mainstream models like the GPT series, one token roughly corresponds to about 4 English characters or 0.5–1 Chinese character, though this depends on the specific tokenizer used. When we say the inference speed is "0.5 tokens per second," it means the model generates one text fragment every two seconds — roughly half an English word. For reference, GPT-4 in the cloud typically outputs 30–80 tokens per second, while llama.cpp running locally on a modern consumer CPU can achieve 10–30 tokens per second.

This speed is essentially unusable for real-time conversation, but from a "can it even run" perspective, it's quite an impressive achievement. Larger models could theoretically be loaded, but inference speed would degrade to intolerable levels. This is because autoregressive language model inference generates tokens sequentially — each step requires a complete forward pass, and inference speed is directly and linearly related to model parameter count and hardware compute power.

Why 90M and Not Smaller?

90M parameters wasn't an arbitrary choice. For conversational tasks, too few parameters cause the model to completely lose linguistic coherence and fail to generate meaningful text. The developer's testing found that a model of this scale still possesses basic question-answering, poetry generation, and short story writing capabilities, albeit with inconsistent quality — it can sometimes correctly answer factual questions like "Which company makes MacBooks" but other times confidently outputs completely wrong hallucinated content.

"Hallucination" refers to when a language model generates factually incorrect or entirely fabricated content with a tone of high confidence. This problem exists across language models of all sizes but is especially severe in smaller models. The fundamental reason is that language models are essentially next-token predictors based on statistical patterns, not knowledge databases. A model's parameters represent compressed storage of statistical patterns from training data — fewer parameters mean less knowledge can be reliably encoded. A 90M-parameter model has roughly 1/2000th the parameters of GPT-3 (175B), so the amount of world knowledge it can store is extremely limited. When the model encounters questions insufficiently covered in its training data, it "fabricates" plausible-sounding answers based on local statistical correlations. For small models, virtually all tasks requiring precise factual recall fall into the "insufficiently covered" category.

This highlights the fundamental limitation of small language models: insufficient knowledge density, weak generalization ability, and extremely high dependence on prompt engineering.

Technical Deep Dive: How LLMPSP Runs an LLM on a PSP

A Minimalist Inference Framework

To run an LLM on a PSP, you must completely abandon all the "luxuries" of modern inference frameworks: no PyTorch, no CUDA, no automatic differentiation, no dynamic memory management. The entire inference stack needs to be rebuilt from scratch, directly operating on raw matrix multiplications and activation functions, with hand-tuned optimizations for the MIPS architecture.

Looking at the GitHub project structure, LLMPSP uses a pure C lightweight inference engine, with model weights quantized and compressed to fit within the limited memory. This shares the same design philosophy as projects like llama.cpp — compressing models to the extreme through quantization and operator fusion — but the PSP version operates under far more stringent constraints.

llama.cpp is an open-source project initiated by Georgi Gerganov, with the core goal of efficiently running LLaMA-series large language models in a pure CPU environment. It's implemented in pure C/C++, doesn't depend on a Python runtime or GPU frameworks like CUDA, and performs matrix operations through GGML (which later evolved into the GGUF format), a custom tensor library. Key technologies include: support for multiple quantization formats (from Q2 to Q8), KV cache optimization, SIMD vector instruction set acceleration (such as x86's AVX2 and ARM's NEON), and memory mapping (mmap) to reduce loading time. This project's success spawned an entire local inference ecosystem, including user-friendly applications like Ollama and LM Studio. However, since the PSP's MIPS architecture doesn't support modern SIMD instruction sets, the LLMPSP developer had to hand-optimize matrix multiplication kernels at an even lower level, making the engineering challenge far greater than porting to x86 or ARM platforms.

Quantization Compression Is Key

90M parameters at FP32 precision requires approximately 360MB of storage — far exceeding the PSP's 32MB of main memory. Aggressive quantization is therefore a necessity.

Quantization is the technique of converting model weights from high-precision floating-point numbers (such as FP32, where each parameter occupies 4 bytes) to low-precision integers (such as INT8 at 1 byte, or INT4 at 0.5 bytes). This process essentially approximates continuous weight distributions through discretization. Mainstream quantization methods include: Post-Training Quantization (PTQ), which directly compresses weight precision after model training is complete; and Quantization-Aware Training (QAT), which simulates quantization error during training to improve final accuracy. More advanced methods like GPTQ and AWQ analyze the importance distribution of weights and apply differentiated quantization strategies across different layers or channels to preserve critical information as much as possible.

INT4 or even lower precision quantization can compress model size by 8x or more, barely fitting it into memory for inference. In the PSP's case, INT4 quantization would compress the 360MB FP32 model to approximately 45MB, but with only 32MB of memory, further compression to INT3 or even INT2 levels may be needed, or a chunked loading strategy must be employed. Each bit reduction significantly decreases the model's representational precision, with the most pronounced damage to long-tail knowledge and complex reasoning capabilities. This extreme compression inevitably introduces accuracy loss, which explains why the model produces obvious hallucinations on certain questions.

Computational Bottlenecks Under the Transformer Architecture

Virtually all current mainstream large language models are based on the Transformer architecture, with Self-Attention as its core mechanism. During inference, each time an autoregressive model generates a token, it must compute attention weights between the current token and all historical tokens, meaning computational complexity grows with sequence length. For a typical Transformer layer, the main computational bottlenecks include: QKV projection matrix multiplications, attention score computation, and feed-forward network (FFN) matrix multiplications.

A 90M-parameter model likely contains 12–16 Transformer layers, each with a hidden dimension of approximately 512–768. Even in this relatively "tiny" configuration, a single forward pass still requires tens of millions of multiply-accumulate operations. For a 333MHz MIPS processor, completing one forward pass takes about 2 seconds — which corresponds exactly to the measured 0.5 token/s inference speed. This computational analysis also clearly reveals why larger models become completely unusable on the PSP: doubling the parameter count roughly doubles inference time.

The Deeper Significance for Edge AI Inference

From PSP to Real-World Edge Deployment Scenarios

The significance of the PSP experiment extends beyond its entertainment value. The current battleground for edge AI inference includes: microcontrollers (MCUs), IoT devices, and low-power sensor nodes. These devices have computing resources on par with the PSP, or even more limited.

The LLMPSP project proves one thing: on a general-purpose CPU without a dedicated AI accelerator, language models with fewer than 100 million parameters can run — at the cost of extremely low inference throughput. This has some reference value for scenarios that don't require real-time responses, such as offline log summarization, local document Q&A, or low-frequency device status analysis.

Comparison with Mainstream On-Device Inference Solutions

By contrast, current mainstream on-device LLM inference solutions are already quite mature:

Apple Silicon: Through the Neural Engine and unified memory architecture, MacBooks can smoothly run 7B or even 13B parameter models
Qualcomm Snapdragon 8 Gen series: Equipped with dedicated NPUs, supporting on-device 3B–7B models on phones
Raspberry Pi 5: With 4GB/8GB RAM, capable of running 1B–3B quantized models at several tokens per second

The NPU (Neural Processing Unit) mentioned here is a hardware accelerator specifically designed for matrix multiplication and tensor operations. Unlike general-purpose CPUs, NPUs can improve neural network inference energy efficiency by orders of magnitude through massively parallel compute units and optimized data paths. Apple's Neural Engine can execute up to 38 trillion operations per second (38 TOPS), Qualcomm's Hexagon NPU achieves 45 TOPS in the Snapdragon 8 Gen 3, and Google's Edge TPU is designed specifically for low-power edge devices, achieving 4 TOPS while consuming only 2W of power. By comparison, the PSP's MIPS processor has a theoretical peak performance of only millions of floating-point operations (MFLOPS) — roughly seven orders of magnitude behind modern NPUs.

The PSP's 90M @ 0.5 tok/s clearly marks the lowest end of this spectrum. It's not a practical solution but rather a yardstick for measuring the lower bound of hardware inference. This enormous compute gap also reveals why the PSP can only run a 90M model while modern phones can smoothly run 7B models — it's not merely a difference in parameter count, but a paradigm leap in underlying hardware architecture from "general-purpose computing" to "AI-specialized computing."

The Ultimate Expression of Local Inference

The original poster said something quite telling: "Doesn't get more local than this." This statement hits on a real tension point in current AI deployment discussions: What does local inference really mean?

From the perspective of privacy protection, offline availability, and data sovereignty, running LLMs locally is a valuable direction. But "local" doesn't mean "without limits" — hardware compute power remains a hard constraint. The PSP experiment demonstrates in an extreme way that even consumer hardware from 2004 isn't entirely excluded from the modern AI ecosystem — the trade-off is just an extremely low capability ceiling and extremely slow inference speed.

The Geek Spirit and Technical Value of the Open-Source Community

This project is fundamentally a geek exploration, and the author themselves admits it's "not practical by any real metric." But the open-source community's enthusiasm for such projects has never waned, for a simple reason: boundary experiments drive cognitive progress.

Similar projects include running neural networks on the Nintendo DS and running small Transformers on Arduino. The common value of these experiments lies in this: they force developers to think about optimization problems under extreme constraints, and the resulting technical insights often feed back into mainstream lightweight inference research. For example, chunked loading and streaming inference techniques developed under extreme memory constraints were later applied to optimize mobile inference frameworks, and experience accumulated with low-precision quantization has also driven the development of mainstream quantization algorithms like GPTQ and AWQ.

LLMSP is open-sourced on GitHub. Interested developers can reproduce it on their own PSP devices or draw inspiration from its minimalist inference engine design for porting to other resource-constrained platforms.

Summary

Running a 90M-parameter LLM on a PSP from 2004, at 0.5 tokens per second with a 1–3 minute response delay — these numbers precisely define the extreme edge of general-purpose CPU edge inference. It has no practical value, but it carries clear cognitive value: it tells us where the boundaries of hardware compute power lie, and what kind of specialized accelerators or more aggressive model compression techniques we need to push beyond those boundaries.