24B Parameters, Only 2.45B Active: Run a Local AI Agent on a 32GB Laptop, 1.7x Faster and Open Source

Running AI agents locally has always come with an awkward trade-off: either the model is small enough to run but too weak, or it's strong enough to be useful but blows up your VRAM. Council (as written in the original), an AI company with MIT ties, recently released an open-source model called 2.5-24 that tries to carve a new path through this dilemma — 24 billion total parameters, but only 2.45 billion activated per token generated, runnable on an ordinary laptop with 32GB of RAM, and it's not designed for chat at all, but specifically targeting local agents.

Why a 24B model can run on a laptop

The answer lies in the Mixture of Experts (MoE) architecture. Think of it as a team of 64 experts rather than a single know-it-all teacher. Each time you ask a question, the system routes it to the four most relevant experts to answer, while the rest stay idle.

So the model holds the complete knowledge of 24 billion parameters, but only a small slice actually participates in computation for each token. This is the origin of "24 billion in knowledge, 2.45 billion doing the work." For local deployment, this design directly determines whether VRAM and memory usage can fit into consumer-grade hardware.

The official quantized versions confirm this: the best-value 4-bit quantized version is 13.5G in size, runnable on a 32GB laptop; the 6-bit version is about 18.3G and requires 48GB of RAM. In other words, most mid-to-high-end thin-and-light laptops from the past two years already qualify.

The core mechanism of Mixture of Experts (MoE)

MoE essentially splits a dense large model into several "expert" sub-networks, paired with a lightweight gating network (router) that decides which experts each token goes to. Traditional dense models must pass through all parameters for every token generated, with computation growing linearly with parameter count; MoE decouples "knowledge capacity" from "per-inference computation" — total parameters determine how much the model can remember, while activated parameters determine the cost of each inference. This is why a model with 24 billion total parameters can compress per-token computation to the 2.45 billion range.

The costs are real too. First, all expert weights must still reside in memory, so VRAM/memory usage is determined by total parameters, not activated parameters. Second, routing itself introduces additional scheduling overhead, and if training is insufficient, tokens may be assigned to experts that aren't well-suited, causing performance fluctuations on long-chain tasks. Common industry improvements include load-balancing losses, expert capacity limits, and shared experts — all aimed at making routing more stable. Understanding this explains why "whether MoE routing is stable in long-chain tasks" is a genuine unknown.

The benchmark numbers actually worth looking at

The easiest way to misjudge this kind of model is by its "overall score" — it doesn't aim to be first in everything. What you might not have noticed is how thoroughly it dominates in the agent and tool-calling direction.

Among the 16 benchmarks published officially, it scores 27.69 on automation tasks versus 8.72 for the runner-up — about a 3.2x lead; 48.09 vs 38.19 on agent testing; 80.71 on tool sandbox, taking first place outright; and first place across all three instruction-following tests. These numbers all point to the same conclusion: it's serious about the chain of "following instructions, calling tools, multi-step execution."

On speed, the official figure is 184 tokens per second. At this level, even for a pure reasoning model — which writes a thinking block to confirm its logic before giving the final answer — users won't feel noticeable lag. It thinks more, but not slow enough to hurt the experience.

Quantization and bit-width: why 13.5G can hold 24 billion parameters

At original FP16 precision, 24 billion parameters require about 48GB of VRAM, which an ordinary laptop simply cannot accommodate. Quantization compresses each weight from 16-bit floating point to a lower bit-width: 4-bit quantization theoretically reduces size to about 1/4, so the article's "4-bit version 13.5G, 6-bit version about 18.3G" follows this pattern. Lower bit-width means smaller footprint and faster inference, but also greater precision loss, manifesting as more errors or repetition in complex reasoning and long outputs.

In actual deployment, memory must also be reserved for the KV cache (which stores context key-value pairs), and the longer the context, the larger the usage — so running 13.5G weights on 32GB of RAM is not a comfortable margin. When choosing a quantization version, it's usually recommended to start with 4-bit, and if you notice rising error rates in instruction-following or tool calling, fall back to 6-bit or 8-bit as a trade-off. This is also the part of evaluating whether a local agent is "good enough" that needs more real-world testing than benchmarks.

What it gives up and where it wins

Treat 2.5-24 as an all-around champion and you'll be disappointed. Hardcore science Q&A, complex code, and fully autonomous web operation are areas where it doesn't excel — territory it explicitly concedes.

Where it wins is very concentrated: it can call tools, perform multi-step execution, follow instructions without drifting, match on math, and the cost per inference is extremely low. This happens to be exactly the core skill tree for local agents — automatically searching for information, reading files, running commands, operating web pages. These tasks don't require the model to know everything, but they do require it to be stable, stay on track, and not burn money.

Its positioning is therefore clear: it's not trying to compete with frontier large models on who's more knowledgeable, but rather to be a specialist in the niche of local agents.

The three most practical things for individuals and small teams

On day one of release, the model had full ecosystem support: Apple chips, cross-platform, and server-side all have corresponding files, pushing the barrier to entry very low.

For individuals and small teams, the value isn't in the leaderboard, but in three very concrete things.

Saving money. Weights are free to download, and once loaded locally, repeated calls incur no additional fees. If local setup feels cumbersome, there's also a cloud API at just a fraction of a dollar per million input tokens — considerably cheaper than most closed-source APIs.

Privacy. Agents need to read your emails, files, calendar, and code — once this data goes to the cloud, it's out of your control. Running locally means data never leaves your machine.

Always available. Agent scenarios often involve high-frequency, fragmented calls. A locally resident model doesn't need to queue or wait for quota — just write and use it.

Function Calling and the agent loop

What distinguishes an agent from a chatbot is its ability to turn model output into real actions. A typical flow: the model decides which tool to call based on the user's goal (search, read file, execute command, operate web page), returns the call parameters in a structured format, the external program executes and feeds the result back to the model, looping through multiple steps until the task is complete. This chain demands not "lots of knowledge" from the model, but "output in format, don't fabricate parameters, and adjust the next step based on returned results."

Therefore tool sandbox benchmarks and instruction-following tests often predict real agent performance better than general knowledge benchmarks. Tool calling failures typically stem from three types of problems: non-compliant formatting, fabricated parameters, and drifting from the original goal after multiple steps. The article emphasizes that 2.5-24 leads in "following instructions, calling tools, multi-step execution," which targets exactly these pain points — but official data still needs third-party reproduction and verification on real tool sets.

A new baseline for local agents

2.5-24 delivers a clear answer for the local agent track: 24 billion in knowledge, 2.45 billion doing the work, runnable on 32GB of RAM, 184 tokens per second, automation benchmark leading the runner-up by 3.2x, free and open source, data never leaves your machine.

It's certainly not the endpoint of general capability — whether MoE routing is stable on long-chain tasks, and whether real-world tool-calling success rates can reproduce the official data, both need more third-party verification. But at least it proves one thing — in the triangle of "cheap, private, always online," local agents finally have a starting point that doesn't require compromise.

One-line Summary

MIT-affiliated team open-sources the 24B-parameter MoE model 2.5-24, targeting local agents runnable on a 32GB laptop.

Paragraph Summary

A team with MIT ties released the open-source model 2.5-24, using a Mixture of Experts architecture with 24 billion total parameters but only 2.45 billion activated per token (4 out of 64). The 4-bit quantized version is 13.5G and runs on a 32GB laptop. It's not designed for chat but specializes in local agents. Among 16 official benchmarks, automation tasks score 27.69 vs 8.72 for the runner-up — about a 3.2x lead — tool sandbox ranks first at 80.71, all three instruction-following tests rank first, and speed reaches 184 tokens per second. Hardcore science, complex code, and fully autonomous web operation are its explicitly conceded weaknesses. For individuals and small teams, the value centers on saving money, privacy, and always available — weights are free to download and data never leaves your machine.