local small model

Local small models refer to AI models typically in the 1B to 10B parameter range that, after quantization and compression, can run on consumer-grade hardware, such as Llama 3.2 3B, Phi-3, and Gemma 2.

The capabilities of current local small models are already sufficient to cover a large number of everyday tasks.

In this solution, the task is merely "modifying a simple set of Python rules in natural language", which places low demands on model capability, so a small local model is entirely sufficient.

Small local models (such as those at the 1B–8B level) can often achieve acceptable accuracy on decision-type tasks, with latency and cost far lower than large models.

Local small models currently struggle with tasks such as the latest news, complex code generation, or high-quality long-form writing.

The advantages of local inference are privacy, offline availability, and low latency, at the cost that model capability is usually weaker than cloud-based large models.

The author suggests that for privacy-sensitive tasks, one should pick a small, private, slightly annoying task, put the smallest local model on it, and keep the numbers in the code while handing the text to the model.

An OpenAI-compatible interface does not mean fully identical capabilities; features that cloud models support, such as function calling, structured output, and long context, may be missing or unstable on small local models.

The author believes the hybrid architecture (sensitive high-frequency tasks on a local small model, complex reasoning on a cloud API) is not a compromise but the correct architecture.

Local small models are limited by VRAM and can usually only run stably in the 8K to 32K context range.

The advantages of local small models lie in privacy and offline availability, but they are limited by phone computing power, which caps the scale and capability of models that can be run.