Building a Self-Improving Team Q&A Bot with Notion Agents

Team knowledge bases are like gardens: neatly trimmed when first built, but overgrown with weeds over time. Notion's official tutorial presents a "knowledge gardening" system made up of three agents that enables a Q&A bot not only to answer questions but also to summon human experts when it can't, automatically draft new documents, and periodically audit outdated content. This system has already been deployed in the sales team of AI company Base 10. This article breaks down its design approach and build methods.

Why Traditional Knowledge Bases Always End Up Abandoned

Notion's presenter Bradford used a vivid metaphor: a wiki is like a pruned garden shrub—well-maintained it's a 3D rabbit, left unattended it's a mess of weeds. He admits this has been the pain point of his entire career—anyone doing enablement work has faced bloated, hard-to-govern knowledge bases. Early content is clean and manageable, but as product lines grow more complex and contributors multiply, updating the wiki falls to the bottom of everyone's priority list, eventually landing on one or two people to clean up endlessly.

This is a Sisyphean task: no matter how hard you try, you can never succeed. The emergence of AI agents made Bradford realize there must be a better way. So Base 10 started restructuring the entire process from the starting point of knowledge retrieval—Slack.

Three-Agent Architecture: Q&A, Archiving, Auditing

Base 10's solution consists of three Notion custom agents, each responsible for a different part of the knowledge lifecycle.

Katz GPT: The "Human-Like" Q&A Expert in Slack

Katz GPT is named after Base 10's first account manager, Anthony Katz. During the team's expansion period, Katz himself was everyone's go-to for correct information: no one knew the product, organization, and customers better than him. Katz GPT's goal is to "clone" this person's brain, turning one person into an infinitely distributable resource through Slack.

This agent connects to multiple data sources including Notion, Slack, Gong, Salesforce, GitHub, PostHog, and the BI tool Sigma. It answers the sales team's questions directly in Slack threads, with answers that are both concise and accurate, and also carry a distinct personality—the tutorial shows colleagues joking with it and asking its opinion of another account manager, and it responds in a tone like "AJ is a solid guy," making the whole interaction more engaging.

Sales Reporter: Archiving Scattered Knowledge Back into the Library

Q&A alone creates a problem: information is output repeatedly but never accumulates. Sales Reporter addresses the second pain point—writing organized and verified information back into the knowledge base. Every day it scans Slack for common discussions, unanswered questions, and emerging topics, automatically drafts Sales Hub documents and places them into a review queue.

The key "human-in-the-loop" design is reflected here: drafts are not published directly but enter a Notion database listing the title, draft page, reason for writing, suggested location, and suggested owner. Only after a human clicks approve does the content automatically move to the corresponding location. Bradford emphasizes that when pushing high-confidence content you can go as fast as possible with minimal friction, but adding new content must include a human review step.

Human-in-the-loop (HITL) is a collaboration pattern in AI system design in which human review or intervention is introduced at key decision points in an automated process, rather than letting the model act fully autonomously. In knowledge management scenarios, HITL is especially important: large language models may generate plausible-sounding but actually incorrect content (i.e., "hallucinations"), and if written directly into the knowledge base it would pollute the information source, misleading subsequent retrieval and Q&A. Sales Reporter's design puts HITL into practice—the agent is only responsible for drafting and recommending, with the final publishing authority left to humans. This division of labor leverages AI's efficiency and coverage while retaining human judgment on accuracy and organizational context. Bradford's emphasis that "high-confidence content can reduce friction, but new content must be reviewed" is precisely a fine-grained control of HITL granularity, striking a balance between efficiency and quality.

Sales Hub Verifier: Periodic Audits to Fix Knowledge Gaps

The third agent audits pages in the Sales Hub that are unverified or missing tags every week, screens out content that needs updating or re-verification, and automatically generates Slack messages to send to page owners. It is precisely this audit mechanism that keeps the vast majority of Base 10's knowledge base content verified and well-tagged.

Building from Scratch: Hands-On with Notion Agents

The second half of the tutorial demonstrates how to build a Q&A agent similar to Katz GPT from a blank slate (named Bradford GPT in the demo). The process can be summarized in a few key actions.

Let AI Help You Write Instructions

Bradford believes that writing instructions is the most daunting part of building a custom agent. His advice is to have Notion AI interview you in reverse: tell it what kind of Q&A agent you want to build and which tools it should connect to, then answer its questions—who the audience is, what style the answers should be in, how to handle weak evidence, and give two or three representative example questions.

In the demo, he deliberately set the agent's speaking style to that of a "salty old pirate" to illustrate how personality settings are reflected in instructions. The instructions Notion automatically generated also proactively added constraints to prevent offensive content and ranked information priority by authoritative documents, verified pages, and Slack discussion timeliness—exactly the retrieval hierarchy he had hoped for.

Instructions, in the context of custom agents, refer to system-level prompts written in natural language that define the agent's role, goals, tone, available tools, information priority, and boundary constraints. Unlike ordinary chat, an agent's instructions need to cover more dimensions: what personality it should speak with, how to respond when evidence is weak, what the retrieval order of different data sources is, and what content should be refused. The quality of instructions directly determines the reliability and consistency of the agent's output. Bradford's suggestion to have Notion AI interview the builder in reverse is essentially leveraging the model's own language capabilities to fill in instruction details humans tend to overlook—such as automatically adding anti-offensive constraints and ranking retrieval hierarchy by authority—thereby lowering the barrier to writing high-quality prompts.

Configuring Triggers and External Tools

Bradford GPT was set to be triggered only in specified Slack channels and to respond only when mentioned. Notion supports direct connections to tools such as Google Workspace and Linear, while other third-party systems can be connected via MCP (Model Context Protocol). In the demo he pasted Sigma's MCP address so the agent could query BI data. After setup, he asked in Slack "What is Base 10's dedicated deployment?" and within seconds received an answer in pirate tone with traceable sources—the whole process took less than five minutes.

MCP (Model Context Protocol) is a standard open-sourced by Anthropic in late 2024 for enabling AI models to connect to external tools and data sources in a unified way. Before MCP, each third-party system (such as BI tools, CRMs, databases) often required separately developed custom connectors and authentication logic, resulting in high integration costs and poor reusability. MCP defines a client-server architecture: the AI application acts as the client, and external systems expose their capabilities (such as querying data or executing operations) by implementing an MCP server, with both parties communicating in a standardized message format. This allows agents to dynamically expand their capabilities like plugging in USB devices. Notion agents support connecting to systems like Sigma that are not natively integrated by pasting an MCP address—precisely through this protocol, avoiding the need to reinvent the wheel for each data source.

Why Put Agents in Public Channels

Some might ask: won't having agents work in public channels make people afraid to ask questions? Bradford's answer depends on organizational culture, but culture can be changed. He recounted the origin of the channel where Katz GPT lives: an account manager first created this "safe space," the president personally liked and supported it, and Katz GPT only entered two months later—after which question volume and participation skyrocketed.

His view is that bringing agents into public forums is a rare opportunity, but only if leadership publicly states "it's okay to be vulnerable here, it's okay to ask questions." This model was later replicated in marketing, recruiting, and deployment engineer teams, each with its own corresponding Notion agent.

Three Pieces of Advice for Builders

Bradford offers several lessons at the end. First, think clearly about what kind of interaction pattern you want with the agent: some run in the background and only reach out when needed; others are triggered in public places because their answers serve multiple people. Interaction design should reinforce organizational culture. Second, use Notion AI to build Notion agents, refining instructions through conversation, testing, and iteration. Third, let real needs guide design—Base 10's entire system stemmed from a simple question: how do you clone Anthony Katz?

The value of this solution goes beyond efficiency. Bradford says that seeing AI and multiple humans collaborating in the same thread is a signal of trust and cohesion—and something knowledge management has always wanted to achieve but never quite could.

Background

Notion Custom Agents is an AI feature launched by Notion between 2024 and 2025 that allows users to build task-specific AI assistants based on Notion pages and databases within the organization as well as externally connected data sources. Unlike general Notion AI Q&A, custom agents can have dedicated instructions, limited trigger scenarios (such as responding when mentioned in specified Slack channels), connections to third-party tools, and independent personality settings. It is essentially a lightweight agent solution that deeply binds large language models with organizational knowledge bases and collaboration tools, configurable without writing code. Base 10's three agents—Katz GPT, Sales Reporter, and Sales Hub Verifier—are precisely a combined application of this capability across the knowledge lifecycle, covering Q&A, archiving, and auditing respectively.