Waku Agent Deep Dive: The Four Core Pillars of a Local-First AI Assistant

From Concept to Code: A Complete AI Agent Implementation

Recently, AI content creator Sean (channel name: Shawn's AI Stories) open-sourced a local-first personal assistant project called Waku Agent. Through a hands-on demo, he brings together the four most critical pillars in today's AI agent systems — Harness, Loop (Loop Engineering), Memory, and Eval — into a single cohesive implementation.

These four pillars represent the mainstream consensus in agent engineering as the field moves from academic research into production practice: Harness defines the toolset, system prompts, and base runtime environment; Loop Engineering is rooted in the ReAct (Reasoning + Acting) paradigm proposed by Google and Princeton University in 2022, where the core idea is to let an LLM iteratively solve complex tasks through a "think–act–observe" loop; the Memory system draws on cognitive science classifications of human memory; and Eval adapts the software engineering concept of Test-Driven Development (TDD) to the domain of LLM output quality assessment.

Unlike many agent tutorials that stay at the theoretical level, Waku Agent's greatest value lies in being actually runnable. It operates similarly to OpenCloud or Hermes Agent, but is designed to be remarkably straightforward: clone the GitHub repo, run it, open the dashboard — and you have a fully functional AI personal assistant running on your laptop. The entire project runs on localhost, embracing a Local-First design philosophy — a principle systematically articulated by the Ink & Switch research lab in 2019, whose core tenet is that data lives primarily on the user's local device, giving users complete ownership of their own data. Samsung's data breach incident — caused by employees pasting internal code into cloud-based AI services — is a classic case that prompted many users to reconsider the value of local deployment. All data is stored locally, nothing is retained in the cloud, and users have full control over their own agent.

The project repository lives under Sean's GitHub account (Shen Shen Chen), named Waku Agent. Interestingly, the project was originally called "Jarvis," but the author realized everyone was building a Jarvis and it lacked originality, so he switched to "Waku Waku" — a Japanese expression for excitement or anticipation — as the voice wake word.

Loop Engineering: Keeping the Agent Thinking Until the Task Is Done

The most intuitive example in the demo is this task: "Find the remaining World Cup matches and add them all to my calendar."

Starting from this simple instruction, we can clearly observe the agent's complete execution logic: the system first checks whether memory retrieval is needed, determines it isn't, and then enters the Loop Engineering phase. The agent calls a search tool and continues cycling — thinking, acting — around the LLM until the task is complete. Users can click in the interface to watch the loop's real-time stream and observe each tool call as it happens.

The theoretical foundation of Loop Engineering is the ReAct paradigm — solving complex problems by alternately generating "reasoning traces" and "task actions." Unlike pure Chain-of-Thought reasoning, ReAct allows the model to dynamically invoke external tools during the reasoning process and incorporate the tool's observations into the next round of reasoning. At the code level, this loop logic lives in the Agent module. By default, the maximum number of iterations is set to 10 — this is known in engineering practice as a "loop guard," a critical safety mechanism that prevents the agent from falling into an infinite loop and consuming unbounded resources due to tool call failures or reasoning drift. Leading agent frameworks such as LangGraph, AutoGen, and CrewAI all implement similar loop control logic under the hood. During execution, the system also displays the cost incurred in real time — the project defaults to the Anthropic API (Claude 3.5 series models are widely favored by agent engineers for their stability in tool calling and coherence in multi-step task reasoning), but users can freely switch to open-source models running via frameworks like Ollama for a fully local deployment with zero marginal cost.

In the end, the agent successfully searched for match schedule information and created a "Norway vs. England" World Cup event on the user's calendar. This is a vivid demonstration of Loop Engineering operating around an LLM agent.

Memory: Three Types Working in Concert

Waku Agent's memory system is the most elegantly designed part of the whole architecture, subdividing memory into three distinct types, each with its own role. This classification maps directly to the long-term memory taxonomy proposed by cognitive psychologist Endel Tulving in 1972 — bringing the cognitive science framework of human memory into AI agent engineering, enabling the system to more precisely manage the storage, retrieval, and update strategies for different types of information.

Semantic Memory

Semantic memory stores persistent facts that are independent of time and place, well-suited for similarity-based retrieval using a vector database. For example, the system records entries like "Sergey is a close friend who loves swimming and often cooks delicious food" and "Raj is a close friend who plays tennis exceptionally well." When a user asks "Who is Sergey?", the agent first passes through a "retrieval gate" to determine whether retrieval is needed, then extracts the relevant facts from semantic memory and returns an answer.

In the demo, the author adds a new memory on the fly: "Vincent and I went to Paris together for a startup interview." The agent recognizes this as a memory storage request, immediately calls the SaveNote tool, and writes this persistent fact into semantic memory. These facts are continuously updated through consolidation tasks.

Episodic Memory

Episodic memory stores dated events, relying more on temporal indexing and date filtering for retrieval. World Cup match schedules and a scheduled meeting with Sam Altman both fall under episodic memory — their defining characteristic is that they are bound to specific points in time.

Procedural Memory (Skills)

Procedural memory stores the tacit knowledge of "how to execute a skill." In this project, it takes the form of Skills — structured Markdown documents injected directly into the system prompt, essentially defining "how you want the agent to behave." For example, a skill called "Schedule Meeting" includes steps like "parse relative dates, check memory context for attendee preferences, call the CreateEvent tool."

The author demonstrates the effect of modifying a skill: after adding the rule "always add a star emoji to calendar events" and saving it, asking the agent to schedule a meeting with Sam Altman produces a calendar event with a star emoji — proof that procedural memory is working. Users can also create new skill files in Markdown format inside the Skills folder. The author even writes a new skill on the spot for "generating YouTube video titles."

Eval and LLM Ops: Observability Is Key for Local Agents

Evaluation System

The Eval folder contains two core directories: Deterministic and Judge.

Deterministic tests validate functionality through explicit rules — for example, checking whether Apple Calendar is working properly or whether working memory is functioning correctly. These tests simply need to answer "did it succeed or not?" For more qualitative questions — such as "is the response quality good? is it healthy?" — the system uses the LLM-as-a-Judge approach. This evaluation method was systematized by Lianmin Zheng et al. in a 2023 paper: for generative outputs whose quality cannot be measured by deterministic rules, a more capable LLM acts as an evaluator and assigns scores. Compared to human evaluation, it is far less expensive; compared to automatic metrics like BLEU or ROUGE, it aligns much more closely with human intuition about "quality." It's worth noting that LLM-as-a-Judge has known limitations: the evaluator model's own biases (e.g., a tendency to favor longer responses) can skew results, which is why Waku Agent combines deterministic tests with LLM-based judgment to form a complementary evaluation system. After the agent replies to the user, the system runs a trace and then an Eval check.

Tracing and LLM Ops

LLM Ops (Large Language Model Operations) is an extension of the MLOps concept into the generative AI era, encompassing tracing, evaluation, prompt version management, and cost monitoring. Observability is borrowed from distributed systems engineering — it refers to a system's ability to fully expose its internal runtime state through three types of data: logs, metrics, and traces.

Clicking the Trace tab, users can see an extremely detailed execution record: how many tokens were consumed, how much it cost, how many LLM calls were made, the specific details of what the retrieval gate fetched from the database, and which tasks took the longest. In the demo, the World Cup query task took close to 100 seconds.

The author emphasizes that all trace information is stored in local files named Traces. This is the essence of the local-first design — tools like LangFuse (one of the most popular open-source LLM observability platforms, supporting recording of token consumption, latency, and evaluation scores per call) or LangSmith (LangChain's official commercial tracing tool) would upload your data to a third party. As a personal assistant, you'd much rather keep that data on your own machine. Of course, if cloud-based tracing is truly needed, the project also supports connecting to Supabase or directly integrating with LangFuse.

Multiple Interaction Channels

Beyond the built-in chat interface, Waku Agent supports multiple interaction modes:

Telegram Gateway: Create a Telegram Bot via BotFather (the author's bot is called "WakuWaku AI Bot") and you can query the agent directly through Telegram, with conversation history synced in real time.

Voice Mode: After running uv run waku, the agent continuously listens for voice input and activates via the wake word "Waku." The author candidly admits that voice recognition accuracy is still not ideal (there were noticeable recognition errors during the demo), and sincerely invites community contributors to help improve the voice component.

System Architecture Overview: One Request's Complete Journey

Putting all modules together, a complete agent execution flow looks like this:

1. Entry: The user sends a request via a gateway (Telegram or chat interface);
2. Memory Preparation: The request passes through the memory system, where a "retrieval gate" decides whether retrieval is needed and which memory type (procedural, semantic, or episodic) to draw from, preparing the correct context for the agent;
3. Tool Calls and Loop: The agent decides which tools to call based on context and runs the full Loop until the task is complete;
4. Eval and Tracing: The Eval system tracks and tests all execution steps;
5. Memory Consolidation: After responding, the agent consolidates new facts into semantic memory.

Additionally, the project manages system prompts via a SOL.md file, allowing users to modify the agent's speaking style in real time through the dashboard. On first use, you'll need to enter the relevant model API keys on the settings page (the author entered keys for Anthropic and Gemini), along with a Tavily API key for web search.

Closing Thoughts: Making AI Agent Design Real

The significance of Waku Agent is that it takes what were previously abstract concepts — Harness, Loop, Memory, Eval — and brings them to life in a real, clean, and fully open-source codebase. For developers who want to understand the full picture of modern AI agent engineering, this is an excellent hands-on learning reference — from ReAct loop control to the engineering implementation of Tulving's memory taxonomy, from LLM-as-a-Judge evaluation to local-first observability design, every module has clear theoretical roots and practical justification.

As the author advocates, this is an open community project — clone the repo, add your own skills and tests, submit a Pull Request. You might just contribute to building the best AI personal assistant running on your local machine.

Key Takeaways