Building a Self-Hosted GitLab AI Review Agent with LangGraph.js: Real-World Lessons from Production

A Code Review Agent Running in Production for Months

A developer recently shared on Reddit their self-hosted GitLab AI Agent built with LangGraph.js. This tool is far beyond a proof-of-concept — it has been running in production for several months, reviewing 1,000+ Merge Requests, and has gone through 133 release iterations since June.

What's even more notable are the real-world numbers: roughly two-thirds of the review comments generated by the Agent were accepted and resolved by MR authors. For an automated tool, that adoption rate signals that its suggestions carry genuine engineering value rather than just creating noise.

The Agent covers three areas of responsibility: reviewing merge requests, handling issues, and providing conversational interaction. The project is open-sourced under the MIT license and is named langgraph-harness.

The Hard Part Isn't the Agent Logic — It's Failure Handling

The author shares a counterintuitive insight: most of the work didn't go into writing the core Agent logic. The real challenge was dealing with the failure modes that emerged during actual operation.

Three typical failure patterns were observed:

• The Agent gets stuck in a loop and fails to converge
• The same tool is called repeatedly, performing meaningless redundant actions
• The run ends with an unresolved question rather than a conclusion

The author's solution wasn't to endlessly tweak prompts, but to introduce small middleware nodes that detect these abnormal states in real time and correct them during execution. He explicitly notes that this approach is "cheaper and more reliable than fixing things purely through prompting."

This is a pattern worth borrowing: rather than relying on the LLM itself to always behave perfectly, build interception and correction mechanisms at the workflow level to keep uncertainty within controllable bounds.

---

LangGraph is an Agent orchestration framework based on graph structures, modeling the Agent's execution as a directed graph: nodes represent specific operations or LLM calls, and edges represent state transition conditions. This structure natively supports loops — the Agent can repeatedly call tools, evaluate results, and decide on next steps until a termination condition is met. However, this flexibility is exactly what gives rise to the failure modes described above: when termination conditions are poorly defined, or when LLM output doesn't match expectations, the Agent may fall into infinite loops or raise new questions when it should be delivering conclusions. Middleware nodes are essentially "guard" nodes inserted into the execution graph — they don't perform business logic, but instead check whether the current state has deviated from the expected path and force corrections or termination when necessary. This approach to ensuring robustness at the orchestration layer is analogous to the circuit breaker pattern in microservices architecture.

---

The Critic Step: Filtering Out Low-Value Comments

Behind the high adoption rate lies a key design decision: a Critic (review) step. Before comments are officially published to an MR, the system first screens the drafted comments and removes lower-value ones.

According to the author's data, this step has discarded roughly one-quarter of all drafted comments so far. In other words, the Agent doesn't dump everything it generates onto the developer — it first runs a round of self-reflection. This is a major reason the two-thirds adoption rate holds up: reducing unhelpful comments is itself a direct way to improve the signal-to-noise ratio.

A Self-Hosted Model Caught Bugs Written by Claude

One of the most interesting details about this project: the review tool reviews its own codebase.

The default self-hosted configuration uses Qwen3.6-35B-A3B (4-bit quantization, deployed via sglang). The author revealed that this locally-deployed model once caught real bugs in code originally written by Claude Sonnet / Opus.

He's careful to note this "isn't a benchmark, just something that happened." While this doesn't constitute a rigorous performance comparison, it sends a clear signal: in a specific code review context, a mid-sized self-hosted model with solid engineering around it is fully capable of catching issues left behind by top-tier closed-source models. For teams that prioritize data privacy and prefer not to send code to external APIs, this is a compelling reference case.

---

Qwen3.6-35B-A3B is a Mixture-of-Experts (MoE) model from Alibaba's Qwen series, with roughly 36B total parameters but only about 3.5B activated parameters — making its inference cost significantly lower than a dense model of the same scale. 4-bit quantization is a technique that compresses model weights from 16/32-bit floating point to 4-bit integer representation, reducing memory footprint to roughly one-quarter of the original with minimal practical accuracy loss, making it feasible to deploy 35B-scale models locally on consumer or mid-range GPUs. sglang is a high-performance serving framework optimized for LLM inference, supporting technologies like continuous batching and RadixAttention, offering better throughput than tools like Ollama for production scenarios with meaningful concurrency requirements. This combination — MoE architecture + aggressive quantization + efficient inference framework — represents the mainstream technical path for practical local LLM deployment today.

---

Per-Project Memory Mechanism

An Agent running long-term faces a practical challenge: how to remain useful after hundreds of runs without repeatedly making the same mistakes or rehashing already-rejected suggestions.

The author's solution is to maintain per-project memory. The system extracts and retains several categories of information:

• Project coding conventions
• Recurring false positives
• Past decisions that have already been made

This memory is deduped and retired over time, preventing unbounded memory growth from degrading output quality. This mechanism is precisely what keeps the Agent useful after hundreds of runs, rather than devolving into a noise generator.

---

The "metabolism" problem of Agent memory maps academically to the knowledge base maintenance challenge in RAG (Retrieval-Augmented Generation) systems, and in engineering terms it closely resembles the cache invalidation problem. If memory only ever grows, coding conventions recorded early on may have since been overturned, false positive patterns may have been fixed in newer versions — yet this stale information continues to be retrieved and influence Agent judgment. Deduplication addresses semantic redundancy — the same convention shouldn't appear multiple times in context, wasting effective context length. "Retiring" typically works through time-based decay or access frequency: memory entries that haven't been hit in a long time are downweighted or deleted. Together, these two mechanisms keep the memory store at a manageable size with high information density, which is the key engineering safeguard preventing Agent degradation over long-term operation.

---

Lessons for Production-Grade LangGraph Systems

This case offers several transferable insights for developers building or planning to build production systems with LangGraph:

Engineering effort should focus on reliability guarantees, not on endlessly refining the Agent's "intelligence." Loop detection, duplicate call interception, and terminal state validation — these seemingly mundane middleware components are the real pillars that keep a system running stably for months in production.

A Critic-style secondary screening can significantly improve output quality. Trading one extra review step for a higher adoption rate is a worthwhile exchange.

Memory metabolism matters just as much — memory that accumulates indefinitely without cleanup eventually becomes a liability. Deduplication and retirement mechanisms are worth considering from the earliest stages of design.

The author closed the post with an open question: have others running LangGraph in production encountered the same issues? This itself is telling — the production deployment of Agents is still in a phase of collective discovery, and honest engineering post-mortems like this one are more valuable than any theory.