Mac Studio Tested: Running a 122B Model as a Household Finance Analyst

Title

Mac Studio Tested: Running a 122B Model as a Household Finance Analyst

Meta Description

Mac Studio M3 Ultra 96GB running Qwen3.5-122B-A10B locally as a household finance analyst. A deep dive into MoE model deployment, fact-checking pipelines, and how to fix LLMs that give confident but wrong advice.

Article Content

Running a large model locally is nothing new, but using a single Mac Studio to let a 122B-parameter model serve as a household finance analyst—and keep it honest—is far more challenging and instructive than "just running a benchmark." In the latest installment of the gharasathi series, blogger koustubh documents his complete journey from a 3B small model to Qwen3.5-122B-A10B. The conclusion up front: bigger models make for prettier explanations, but arithmetic safety must be backstopped by code.

Hardware and Model: Why Mac Studio Plus MoE

The entire system runs on a Mac Studio with an M3 Ultra and 96GB of unified memory. The model is the 4-bit GGUF quantized version of Qwen3.5-122B-A10B (UD-IQ4_NL), which occupies about 62GB of memory in practice, leaving 30-plus GB to simultaneously run Immich, Nextcloud, and a Kubernetes cluster. This "one machine does it all" deployment is exactly where the unified memory architecture proves its worth.

The core reason for choosing this model is its Mixture-of-Experts (MoE) structure: 122B total parameters, but only about 10B activated per token. All experts must reside in memory, and unified memory happens to excel at this; meanwhile, the compute cost per token is close to that of a 10B dense model. The result is "big-model quality at acceptable speed"—a dashboard follow-up question with tool calls takes roughly 25 seconds, while overnight batch analysis is more than sufficient.

The server does not use Ollama; instead it runs llama.cpp's llama-server directly, because precise control over context length, parallel slots, chat templates, and the thinking toggle is required. Nix pins the build version, and launchd handles process supervision. Context is set to 128k tokens with 2 parallel slots, exposing an OpenAI-compatible API.

One notable detail is thinking-mode management. Qwen3.5 thinks by default, but the server disables it by default, leaving each caller to enable it as needed: the overnight analyst turns thinking on, while chat and routing leave it off. The reason is that thinking has no natural stopping point on hard prompts—the author has seen a single call burn through 35k tokens—so all thinking requests must be capped.

Mixture-of-Experts (MoE) is an architecture that splits the feed-forward network inside a Transformer into multiple "expert" sub-networks. Each token is routed through a gating mechanism to only a few experts for computation, so the total parameter count can be very large while the floating-point operations per inference are far lower than a dense model of equivalent total size. Take Qwen3.5-122B-A10B: 122B is the sum of all expert parameters, and A10B means about 10B parameters are activated per token on average. The cost of MoE is that all expert weights must reside in VRAM or memory at all times—unlike dense models, you can't load only some layers—so memory capacity requirements are higher. In exchange, it delivers near-large-model quality at near-small-model speed on the same hardware. This is also why unified-memory Macs are particularly well suited to running MoE models.

Two Use Cases: Chat Assistant and Overnight Analyst

The first use case is a chat agent named anna, built on LangGraph and wired up with a dozen or so financial tools: spending analysis, insights, goals, a mortgage what-if calculator, bills and statements, and a read-only Cypher query. Its prompt rules are blunt: it must call a tool before answering, never do arithmetic in its head, and route all calculations through the calculate tool. On the dashboard, users can click a fixed question and then keep asking follow-ups; follow-ups carry the conversation context, so a question like "the loan is fixed—does that change the conclusion?" gets a specific answer about that finding.

The second use case is where the large model truly earns its keep: the overnight analyst. The pipeline has several stages. First comes the fact pack, built by code by topic—covering mortgage and offset accounts, childcare, utilities, groceries, dining, insurance, and so on—with each pack containing monthly totals, merchants, recurring charges, balances, and facts read from bills and statements. The key principle: every number in the pack is computed, not estimated.

The model then summarizes 3 to 6 themes for the household, then writes up to 3 findings per theme. Before writing, the model can request supplementary data from a fixed list—say, transaction records for a given merchant or month-by-month data for a category. What happens after writing is the heart of the whole system.

Fact-Checking: Making the Model Afraid to Make Up Numbers

Every amount and percentage in every finding must be traceable to a source in the fact pack, or derivable from the pack through one-step arithmetic. Findings that fail get one round of repair; fail again and they're rejected outright. The original post includes last night's log: one finding about a childcare absence allowance was rejected because "the savings amount of 156.0 is not in the data"; another about the CCS subsidy rate was withdrawn by the analyst after fact-checking. Both sounded perfectly reasonable but were numbers the model had computed itself.

After fact-checking comes a second judgment: another round of calls asks whether the advice actually follows from the facts. Then comes cross-theme synthesis, ranking based on historical feedback ("never used," "already handled"), merging duplicates, and capping the number of new findings per night.

The author emphasizes that every finding stores its evidence: transaction IDs, or quoted lines from PDFs. The fact-checker works against this evidence. This becomes even more critical when processing bills and statements later on.

Mistakes LLMs Still Make: Confident but Lacking Context

Fact-checking can stop fabricated numbers, but it can't stop advice that's out of context. The author shares a personal experience: the analyst told him his mortgage rate was less than one percentage point above a "previously negotiable rate" and suggested calling the bank. Every number passed verification, but the advice was useless.

There are three layers to why. The loan is fixed-rate, so repricing means break costs; the loan-to-value ratio (LVR) means switching banks requires paying lenders mortgage insurance (LMI) again, enough to wipe out years of savings from a small rate reduction; and the so-called "negotiable rate" came from an earlier finding and wasn't in the data at all.

The fix has two parts. First, add domain rules to the fact pack: break costs on fixed-rate loans, LMI when LVR exceeds 80%, never citing rates that don't exist in the data, and the real levers—offset account balances, retention rates when the fixed term ends, and refinancing once LVR drops below 80%. Second, add "household facts"—information only the residents know, such as LVR—provided as brief "household notes" to both the analyst and anna.

After the fix, when the author asks whether refinancing is realistic, anna says no, explains LMI and break costs, and points to the offset account. And the offset account is where the real gold in the data lies: in the months with the highest offset balances, daily interest dropped by about a fifth.

Four Lessons for Anyone Trying This

The author's takeaways are worth going through one by one. First, bigger models make for better explanations but don't make arithmetic safe—no matter how big the model, counting logic belongs in code. Second, enforce citations: every finding stores its evidence. Third, feed the model the constraints you'd tell a human advisor—most bad advice comes from missing context, not insufficient reasoning. Fourth, unified memory is the enabler for all of this: a 62GB 4-bit 122B MoE model still leaves plenty of room for other services.

The next post in the series turns to bills and statements—where the author deliberately didn't use an LLM, only to find they held the most valuable findings. That reversal is itself worth noting: in scenarios like financial analysis, deterministic code paths are often more reliable than generative models.

Background

GGUF quantization is the model file format used by the llama.cpp ecosystem, supporting a range of quantization schemes. The UD-IQ4_NL used here is a 4-bit asymmetric quantization; "UD" stands for Unsloth Dynamic quantization, which applies different bit widths to different layers or tensors, keeping sensitive parts at higher precision to minimize quality loss at an average bit width of about 4 bits. The direct benefit of 4-bit quantization is memory footprint: 122B parameters at 4-bit take roughly 60GB, versus more than 240GB at FP16, allowing a single 96GB unified-memory machine to hold the entire model. Quantization does bring a slight quality drop, but with fact-checking and code backstops in place, the author considers it entirely acceptable.

LVR (loan-to-value ratio) and LMI (lenders mortgage insurance) are key to understanding why the refinancing advice in the article fails. LVR is the loan balance as a proportion of the property's valuation; when LVR exceeds 80%, lenders typically require borrowers to buy LMI to protect the lender in case of default. LMI premiums can run from one to several percent of the loan amount and are usually paid upfront or added to the loan. When a borrower refinances from one bank to another, if the new loan's LVR is still above 80%, LMI often has to be paid again—a cost that can wipe out years of savings from a small rate cut. In addition, paying down or refinancing a fixed-rate loan during the fixed term usually incurs break costs, the fee the bank charges to compensate for its lost funding cost. These two constraints are exactly why the model's seemingly reasonable "call the bank to negotiate your rate" advice was in fact unworkable.

One-line Summary

Running a 122B MoE model locally on a Mac Studio for household finance analysis, with code backstopping arithmetic and fact-checking.

Paragraph Summary

Blogger koustubh built a household finance analyst on a Mac Studio with an M3 Ultra and 96GB unified memory, using a 4-bit GGUF quantized Qwen3.5-122B-A10B. The MoE architecture lets a 122B-total-parameter model activate only about 10B per token, balancing quality and speed. There are two use cases: the chat agent anna, which is forced to call tools and forbidden from doing mental arithmetic; and the overnight analyst, where code builds fact packs and every finding's amounts and percentages must be traceable or be rejected or sent for repair. Fact-checking stops fabricated numbers but can't stop out-of-context advice—such as refinancing suggestions that ignore fixed-rate break costs and LMI. The author's conclusion: bigger models explain better, but arithmetic safety must be guaranteed by code, and every finding must store its evidence.