SoloOps AI: Using Local AI to Catch Clients Sneaking in Free Work

Independent developers dread hearing one sentence: "Can you just add WhatsApp OTP login? Customers keep forgetting their passwords." It sounds like a small change, you nod and build it, but you never send an invoice. This is scope creep, and it's the most common way freelancers lose money. The open-source project SoloOps AI tries to plug this hole with an auditable workflow: it binds every piece of work to approvals and invoices, and uses a locally running AI model to judge whether a new request is already covered by confirmed requirements.

Product positioning: a "leak-proof" console for freelancers

SoloOps AI's core pipeline is a traceable state machine: client → project → requirement → change request → approval → task → billable work → invoice. When a client message comes in, the system calls the local open-weights model qwen2.5:7b (deployed via Ollama) to compare the message against the requirements the client has already agreed to, answering just one question: is this new paid work, or is it already included in the contract?

The key point is that the AI cannot reach a conclusion on its own. It must cite the exact wording of both the client message and the confirmed requirement as evidence, and then a human decides. In the demo, the author shows two typical scenarios: a client asks "Customers will log in with email and password, right?" The system cites REQ-1 and marks it as "possibly covered"; whereas "add WhatsApp OTP login" is marked as "new scope" because no confirmed requirement matches.

By the author's own estimate, at 1,500 Indian rupees per hour, four "quick additions" of three hours each per month add up to 18,000 rupees of unpaid labor a month. SoloOps itself is MIT-licensed, costs nothing, has no paid tiers, and the AI runs on the local machine with no API bills.

The most dangerous thing isn't a weak model — it's an overconfident one

The most valuable part of the project isn't the features themselves, but the author's honest assessment of the AI's capabilities. He fixed the thresholds before running the evaluation, and the most critical metric, T4, is the rate of "incorrectly judged as covered," required to stay under 5%. Because once the AI says new work is included, the developer works for free — a direct financial loss.

The first round of testing (23 synthetic cases × 3 runs) did not look good: qwen2.5:1.5b had only a 30% correct-judgment rate, and qwen2.5:7b managed only 46% — both failing. More counterintuitively, the larger model was more dangerous — qwen2.5:7b told the developer "already covered" incorrectly in 14% of cases.

One real example: a client wrote "per contract REQ-9, add gift wrapping at checkout," but no REQ-9 existed in the project, and there was no gift-wrapping requirement either. The client just sounded confident, and qwen2.5:7b answered "covered" in 2 of 3 runs. Without an extra check, the developer would have built the gift-wrapping feature for free because the AI believed a confident client.

The "covered" judgment that recurs throughout the article is a specific state in scope management: a request raised by the client is already included in some confirmed and priced requirement in the project. The opposite is "new scope," i.e., work that needs a separate quote, approval, and invoice. The reason this distinction matters is that verbally confirmed "requirements" tend to be scattered across emails, chats, and meeting notes without structure, leaving developers to judge from memory — which is exactly how requests like "just add this quickly" most easily turn into unpaid labor. Making the coverage judgment an explicit binary classification problem is what allows the model's error rate to be quantified, and T4-style loss-oriented thresholds to be set accordingly.

The deterministic evidence gate: let code, not the model, be the final judge

Rather than swapping in a bigger model, the author changed the architecture. The model is only responsible for breaking the client request into verbatim-cited "claims," each accompanied by a verbatim quote from a confirmed requirement. Then SoloOps' TypeScript code checks, one by one, whether each citation actually exists in the stored text. A fabricated requirement number, a paraphrased quote, a draft-status requirement, or a contradiction will downgrade the result to "unclear" and show why.

This design performed remarkably well on the holdout set. The author wrote 32 holdout cases and committed them before building the gate, pre-declaring how they would be run. The results showed that qwen2.5:7b's own judgment had a 41.7% false "covered" rate on the holdout set, which dropped to 4.8% after the SoloOps gate — passing all five thresholds. Back to the gift-wrapping case: after the gate, the model still said "covered" in 2 of 3 runs, but it couldn't cite any real requirement related to gift wrapping, so SoloOps marked all 3 runs as new scope. The model's opinion is still displayed, but it no longer has decision authority.

The author also admits that the gate can't solve everything. In case X01, two confirmed requirements contradict each other: one says guests can check out, the other says an account is required. The model correctly cited one of them, and the citation was real, but it still judged "covered" in all 3 runs. Verbatim validation cannot detect conflicts between two real citations. So the README's conclusion is blunt: no model is currently recommended, and contradictory requirements remain a known failure mode. This is precisely why the product positions the AI as "advisory only" — if it's wrong, the result shows as unclear, or a human corrects it, and nothing can be approved or invoiced without human confirmation.

A "deterministic evidence gate" is an architectural pattern for post-validating LLM output: the model can generate conclusions freely, but its conclusion is only accepted when a set of model-independent rules all pass; otherwise it is downgraded to "unclear." Deterministic here means the same input always yields the same judgment, as opposed to the randomness introduced by model sampling. Verbatim citation verification is one concrete implementation — exact string matching of the model's quotes against the original text of confirmed requirements in the database. Similar ideas appear in RAG citation tracing, rule-engine approvals in finance, and "LLM proposes, symbolic system verifies" hybrid architectures. The cost is that only form can be validated, not semantic-level contradictions.

The significance of open source and local deployment

SoloOps chose a local open-weights model for more than cost savings. Client messages and project requirements are confidential, and local operation means data never leaves the laptop, no third-party AI API is needed, and no data processing agreements need to be handled. For freelancers in markets like India, a zero-cost tool lowers the barrier to protecting their income.

Technically, any OpenAI-compatible endpoint can be plugged in. The evaluation code and all raw run logs are in the repository, so anyone can rerun them on a new model and publish the numbers rather than trusting vendor marketing. The safety logic is written in readable TypeScript rather than hidden behind an API. The author also acknowledges that large hosted models would likely outperform a 7B model on a laptop at scope judgment, and wouldn't need a GPU, but he still wouldn't let one decide on its own, because the root cause of the gift-wrapping problem is the client's confident tone, not the model's size. The choice is left to the developer: those willing to send client text can plug in a larger model, running npm run eval:ai first to see the numbers before deciding.

Ollama, mentioned in the article, is a local model runtime that downloads, loads, and exposes open-weights models as an HTTP service, with an OpenAI-compatible interface style, so an application only needs to change a base URL to switch backends. The 7b in qwen2.5:7b refers to roughly 7 billion parameters, a scale that can run on consumer GPUs or laptops with unified memory; fewer parameters means faster inference and lower VRAM requirements, but usually weaker semantic judgment. A holdout set refers to test data that is never looked at during tuning and design and is evaluated only once at the end; the author committed the cases before running them precisely to avoid the optimistic bias caused by reverse-tuning on the test set.

Engineering implementation and quality assurance

The project uses Next.js 16, React 19, Hono, a MongoDB replica set, Zod, and TypeScript, managed with npm workspaces. Amounts are calculated as integer paise and all on the server side; approvals are bound to immutable versions and their SHA-256 hashes; only the owner can issue invoices, and invoice numbers are sequential with no gaps. The ai module has no path that writes important data — the AI never proposes, approves, prices, or invoices anything.

On testing, the project includes 1,137 unit, component, and API integration tests running against a real MongoDB replica set, plus 34 Playwright end-to-end tests. The author used Claude Code to assist development, constraining behavior changes through project skills, security reviews, and read-only review agents, and requiring that every behavior change update requirements, ADRs, and the traceability matrix in the same commit. The repository's CLAUDE.md lists invariants the agent must never break, such as "AI advises, humans decide."

Takeaways worth noting

The value of SoloOps AI goes beyond being a freelancer tool. It demonstrates a pragmatic pattern for handling AI uncertainty: don't let the model make the final call; let the model provide verifiable evidence, let deterministic code check whether the evidence holds, and let a human decide. This pattern can be transferred to contract review, expense auditing, compliance checks — any scenario where "an AI misjudgment causes real financial loss."

The project is currently in release-candidate status. M1–M5, along with workspace members and the AI scope-check feature, are implemented and have passed the local test suite, and CI is green. What remains is manual verification by running the dev server and a browser. For readers interested in local AI deployment, evidence-gate design, or freelancer workflows, this repository is worth a close read.

One-line Summary

An open-source tool uses local AI plus a deterministic evidence gate to stop freelancers from working for free due to scope creep.

Paragraph Summary

SoloOps AI is an auditable workflow for independent developers that strings clients, projects, requirements, change requests, approvals, tasks, and invoices into a traceable state machine, and uses a locally running qwen2.5:7b model to judge whether a new client message falls within confirmed scope or is new paid work. The author fixed evaluation thresholds before testing and found that the larger model was actually more dangerous, with a 14% false "covered" rate that would cause unpaid labor. So he changed the design to let the model only provide verbatim citations, with TypeScript code verifying their authenticity, cutting the false coverage rate on the holdout set from 41.7% to 4.8%. The gate cannot catch contradictory real requirements, so the AI is advisory only and humans hold final decision authority. The project is MIT-licensed, has zero API cost, and all evaluation code and raw data are open source.