Local Dating Chat Copilot: 322M Judgment + Gemma Drafting, Data Never Leaves Your Computer

When They Reply With Just Two Words: "I'm Fine"

You type, delete, then type again—three versions of a reply sit in the input box. Is it really fine, or do they want an apology, an explanation, or just some space? A more practical question: whatever you send next, would you really want a cloud service to read through the entire conversation first?

This is exactly the problem Laya-Heart aims to solve. It's a "dating chat copilot" that runs on your local machine: copy their message, switch to the page, Ctrl+V, and within half a second a small card appears below the message, using probabilities to offer 1–3 follow-up questions ("Is 'fine' really fine? What does the other person need right now?"), or a danger level from 1 to 10. Meanwhile, Gemma 4 E2B immediately starts drafting two candidate replies, with the first completing in about 3.6 seconds on a laptop CPU. You pick one, tweak it, hit Enter to copy, and paste it back into your chat app yourself.

It doesn't connect to any chat app, and it never sends messages for you. By default, what you paste never leaves your machine.

Division of Labor Between Two Models: First Read the Subtext, Then Write the Reply

The entire system is powered by two open-source models with clear division of labor.

Laya: A 322M-Parameter "Read the Room" Judge

Laya-multilingual comes from Convai Innovations, with 322M parameters, built on mmBERT, Apache-2.0 licensed. Its job isn't to generate text but to judge: providing probabilities for each card's questions, then answering a hidden safety question. The author has released ONNX weights for both the zero-shot and fine-tuned v3 versions on Hugging Face.

Gemma 4 E2B: Responsible for Drafting and Memory Filtering

Gemma 4 E2B writes two candidate replies and picks out facts from the chat worth remembering. The author did LoRA fine-tuning, exported Q4_K_M GGUF (3.4 GB), and runs it on CPU via llama.cpp's llama-server.

A detail on how "memory" works: your chats with each person are stored locally, and you can also write a small "book" for the other person (nickname, landmines, the day you met). When a new message touches content in the book, matching sentences are passed to Gemma as context. The app also automatically saves facts from the chat, using the original wording, which is undoable, and parses relative dates—"Next Saturday (10-24) is my birthday" gets pinned to a specific date.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: all original weights of the pre-trained model are frozen, and only a small number of trainable low-rank matrices are inserted next to certain layers (rank 16 means the low-rank decomposition dimension). Trainable parameters typically account for only a tiny fraction of the original model. Its advantages are fast training and low VRAM usage—a round can complete in tens of minutes on a free Kaggle T4, and the resulting adapter file is small; at inference time it can be merged back into the original weights or loaded separately.

GGUF and Q4_K_M are the model file format and quantization scheme used by the llama.cpp ecosystem. GGUF packages weights, tokenizer, and metadata into a single file for easy distribution; Q4_K_M means weights are stored at roughly 4-bit precision, where K stands for block-wise mixed quantization strategy and M is the medium tier—key layers retain higher precision while others are compressed more aggressively. Compared to fp16, model size shrinks to about a quarter, which is the prerequisite for the 3.4 GB Gemma 4 E2B to run at acceptable speed on a laptop CPU.

Making a 2B Model "Feel Fast" on a Laptop CPU

This is the most engineering-valuable part of the article. On a laptop CPU, Gemma 4 E2B reads prompts at about 55 tokens/s and writes at about 12 tokens/s, so wait time is mainly spent on reading.

Initially, the author called Gemma through Ollama's chat API, which only reuses the sliding window cache when the prompt is exactly identical. So each new message required re-reading the entire 400–475 token prompt, taking 6.4–9.4 seconds. Three changes cut most of the waiting:

• Start llama-server yourself with --swa-full to retain everything already read;
• Reorder the prompt structure for caching: fixed rules first, chat history only grows from the tail, and a short tail at the end converting Laya's answers into instructions. New messages now only require reading 62–105 new tokens instead of 400+;
• Start drafting as soon as the card completes, stream output, and stop once two replies are written.

Actual measurements (Intel i5-13420H, 16 GB RAM, no discrete GPU): card median 0.44 seconds, first complete reply 3.6 seconds, both replies 5.8 seconds. This was tested on a busy machine with about 3 GB of memory used in the background; on an idle machine with the original models, the times were 0.39 s, 3.5 s, and 5.0 s respectively, with the two models using about 2.3 GB of memory combined.

First launch only requires downloading a ZIP and double-clicking start.bat, then clicking one button to download about 4.8 GB of model files, with resumable downloads and fixed SHA-256 verification for each file.

Sliding Window Attention cache (SWA cache) is the root of the performance problem here. Models using sliding window attention only allow each token to attend to a fixed-length range of preceding context, so inference frameworks by default only need to retain the KV cache of the most recent window, discarding earlier content. The benefit is constant memory usage for long contexts; the cost is that when a new request's prompt differs from the last only in the tail, the framework cannot reuse the entire prefix and must re-read it. --swa-full makes llama-server retain the full cache rather than just the window portion, thus enabling prefix reuse.

KV cache and prompt prefix reuse refers to: when an autoregressive model processes a prompt, it computes and caches the key-value pairs needed for attention for each token. If a new request shares an identical prefix with an old request, the framework can directly reuse these caches and only compute new tokens. This is why prompt structure places fixed rules first and lets chat history grow from the tail—any change in the middle invalidates all subsequent caches.

Fine-Tuning the Judge: F1 from 0.33 to 0.57, Safety Recall from 0% to 62%

Out-of-the-box Laya is a general-purpose judge, not tuned for dating chats. The author built a training toolchain and ran the full pipeline.

Training data relies on two "teachers": Teacher A (DeepSeek^Ad-V4.1-Flash) writes 3000 practice conversations to fit selected answers, then reads each conversation five times per question, discarding those that don't match expected answers; Teacher B (Qwen3, a different model family) writes and labels another 300-conversation practice exam to ensure the model isn't graded by the teacher that taught it. Training runs on a free Kaggle T4, about one hour for three random seeds, with thresholds set before training.

300-conversation practice exam (Teacher B: Qwen3-32B)	Zero-shot Laya	Fine-tuned v3
Macro-F1 for 11 questions	0.33	0.57
Calibration error (ECE)	0.15	0.07
Danger level mean absolute error (1–10)	2.78	1.56
Safety recall at 0.8 alarm threshold	0/21	13/21 (62%)
False positives on normal chats	2.9%	1.4%

The 95% confidence interval for the F1 improvement is 0.18 to 0.31. The author candidly admits two thresholds weren't met: safety recall of 62% against a target of 90%, with 5 of the 8 missed risk conversations being "control and abuse" read as no-risk, such as "You'll never see your sister again unless it's through me"; and the "has the crisis passed" question dropped 0.08 in F1, against a limit of 0.02. However, v3 raised the app's safety alerts from triggering on 4 of 21 risk chats to 13, so the app has used v3 since October 18, with a held-out exam deciding its fate.

The entire fine-tuning round cost about 23 RMB (about $3) in DeepSeek API calls, plus about 9 Kaggle weekly free GPU hours. Running the full toolchain also uncovered 41 issues in code and documentation, including a caching bug that silently turned 20% of practice conversations into near-duplicate samples, fixed before final training.

Three Valuable Fine-Tuning Lessons

• Compare averages, not the best seed. F1 varies by 0.02–0.05 across random seeds for the same version; averaged over three seeds, v1, v2, and v3 were 0.57, 0.56, and 0.56 respectively.
• Adding more risk samples bleeds into other answers. Raising the proportion of risk conversations from 10% to 25% (v2) increased recall on an independent safety check from 53% to 68%, but the model began judging more chats as emergencies, missing partners who were "just angry." v3 trained risk samples only on the safety question: recall reached 79% with no false positives on 47 new risk chats + 19 normal chats, and side effects largely disappeared.
• You can't test what your teacher can't write. Having Qwen3-14B write 25 risk exam items yielded only 5 with clear signals; switching to stricter prompts and the larger Qwen3-32B brought the exam to 21 items.

Macro-F1 first computes the harmonic mean of precision and recall (F1) for each class separately, then takes an unweighted average across all classes. Unlike micro-F1, which weights by sample count, macro-F1 gives rare classes the same weight as common ones, so poor performance on minority classes is amplified—exactly the metric needed for safety-related questions. When the 11 questions have uneven option distributions, a model that always guesses the majority class will have a low macro-F1, and 0.33 roughly reflects that level.

ECE (Expected Calibration Error) measures whether the probabilities a model gives are trustworthy: predictions are bucketed by confidence, and the average confidence in each bucket is compared with actual accuracy, then the absolute differences are weighted and averaged. An ECE of 0.07 means when the model says "70% likely," the actual occurrence rate deviates by about 7 percentage points on average. For a system that triggers safety alerts, calibration matters more than raw accuracy—the alarm threshold is set at 0.8, so if probabilities are systematically too high, false positives surge; if too low, alerts are missed.

Fine-Tuning the Writer: 133 Wins, 67 Losses, Fabrications Dropped from 16 to 1

Gemma was also fine-tuned, and the app now downloads the fourth of four runs, with the choice determined by rules written before training.

• Data: 1496 samples. The Chinese portion was written by Teacher A following the app's exact prompt to produce candidate reply sets, with the app's rules filtering out bad sets and the highest-scoring becoming training replies; 108 English replies were the original Gemma's own drafts, similarly filtered and scored, so all stay within the app's 60-character limit.
• Training: A 47-minute job on Kaggle T4, training LoRA with Unsloth (rank 16, two epochs), exporting a 3.4 GB Q4_K_M GGUF.
• Results: On all 297 practice exams, Teacher B (Qwen3-32B) blind-evaluated in two orders, comparing the fine-tuned version against the original Gemma: 133 wins, 67 losses, 97 ties (counting ties as half gives 61%, and 65% on Chinese chats). The fine-tuned version fabricated or over-promised once, versus 16 times for the original Gemma.
• Still one threshold unmet: In the toolchain's 50-item self-check, all thresholds passed except the win rate of 0.54, below the 0.55 threshold (a second evaluation on the same 50 items gave 0.57).

Why Open Source Wins Here Over Closed Source

The author used a very concrete comparison: he built on Jev Chat Assistant, which by default calls a closed-source hosted judge and sends the last ten lines of chat to it. That model can't be fine-tuned, and the terms don't permit using its answers to train models. Laya's weights are open, so it can be fine-tuned for dating chats against self-defined standards, and it runs on the user's own computer, sending 0 bytes.

These points deserve elaboration:

Verifiable privacy. Dating chats are arguably among the most private data, and the other person never agreed to have their messages sent anywhere. Open weights make this possible: both models run locally, the server only listens on 127.0.0.1, and a badge in the page header counts bytes sent out per round (zero by default). Users' concerns are real—Pew Research shows 71% of US adults believe AI will make their personal information less secure.

A cost structure that allows one-time pricing. There's no per-message billing; the entire fine-tuning round cost about $3 plus free GPU hours, so the app can be sold at a one-time price rather than a subscription. Based on public census and survey data, the author estimates about 39–51 million people in the US aged 20–34 who have partners and use smartphones, of whom 17–27 million have access to a Windows PC; the same measure for China is 138–162 million and 41–94 million. Comparable mainstream apps (RIZZ, WingAI, YourMove, etc.) sell for $29.99–79.99/year on the US App Store, and he plans to charge a one-time $14.99 (68 RMB in China), keeping the code MIT-licensed.

Tunable for narrow scenarios. Reading subtext in Chinese dating chats ("You'd better," "I'm fine," "Whatever") is a very narrow task that can be handled by training a small judge with your own data against your own thresholds; when results are surprising, you can open up the model and see why—seed noise, data mix, or problems with the exam teacher.

No lock-in. Settings allow pointing to another GGUF file or Ollama model, or to a laya-serve endpoint.

Safety Rules Written in Code, Not Left to the Model

After each card, Laya also answers a hidden safety question, and the code checks chat keywords simultaneously. Signs of self-harm or being abused/controlled trigger alerts; in self-harm cases, candidate replies include a human-written option not generated by Gemma. If the chat appears to involve minors, nothing is drafted until both parties are confirmed to be adults. On the 300-conversation practice exam described below, 13 of 21 risk chats triggered alerts (keyword rules alone catch 4), with a 1.8% false positive rate on normal chats.

It should be noted: Laya-Heart only drafts replies; it is not psychological counseling or crisis support. In a crisis, contact local emergency services or a helpline. Also, the other person may not agree to having their messages analyzed, so use with care.

A Record of Using an AI Agent to Investigate a Tracking Leak

The author used Claude Code to complete most of the code, training, and demo recording. One debugging session in the agent conversation is particularly representative: building a "numbers only" guard for Sentry tracing.

After writing the tracing module and 7 unit tests, the agent didn't stop at self-testing. It pointed a fake DSN at a localhost receiver, ran a round with the real Sentry SDK, and checked byte-by-byte what would be sent—discovering that Sentry 11 streams spans individually, so the scrubbing hook it had written for the entire transaction never executed. No data leaked, but only because the spans contained no text to begin with. It moved the guard to the correct location and verified both directions: a span stuffed with chat text arrived with only token counts remaining; removing the guard made the same check fail and print the leaked words.

Running for real exposed two more bugs in tracing itself: the streaming issue above, and the first real run becoming a single 194-span trace, because with the default integration disabled nothing opened a trace per request. Fixed with Sentry.startNewTrace at the start of each round. A 40-round trace with the fine-tuned model also showed: three Laya calls in one round (two card questions plus the hidden safety question) took 1.25 seconds on a busy CPU before Gemma could start; the AI Agents panel counted 288 Laya calls and 40 Gemma calls totaling 51,602 tokens, with estimated cost remaining $0—because both models run on the laptop.

This process itself demonstrates the value of open source + local inference: if the models were in the cloud, this kind of "byte-by-byte confirmation of no leak" verification would be impossible.

Sentry's trace and span are a two-level structure in observability: a complete request chain is called a trace, and each operation in the chain (a model call, a database query) is a span; spans can be nested and record metadata such as duration and attributes. Sentry 11's SDK sends finished spans individually in a streaming manner by default, rather than submitting everything at once after the entire trace ends.

This explains why the scrubbing hook written for the entire transaction didn't execute: the hook was attached to the wrong lifecycle point, and spans were already sent before the hook ran. The correct approach is to attach scrubbing logic to the callback before each span is sent. This also shows that "no data leaked" may have been accidental—if span attributes happened to contain text, a failed hook would have sent it out directly, so you must use the real SDK against a local receiver for byte-by-byte verification and run negative tests that should fail when the guard is removed.

Try It

The online demo (stand-in models, sample data) lets you experience the full flow of the real page in your browser: paste a message, pick a reply, edit it, add a contact—nothing leaves the tab. To run the real models, download the ZIP and double-click the launch script; the first click downloads about 4.8 GB. Evaluation numbers can be reproduced with one command: npm run reproduce takes about two seconds to offline-recompute the 62 evaluation numbers in the report; adding --models downloads both Laya models at fixed versions and reruns 68 checks on your CPU, with each probability within 0.001 of the recorded run.