Crew: Enabling Multiple AI Agents to Collaborate and Share Knowledge Like a Team

From Solo Combat to Team Collaboration

Most AI programming assistants today operate in a "solo combat" mode—each agent completes tasks independently, and the experience it accumulates dissipates once the task is done.

An AI agent refers to an AI system capable of perceiving its environment, making autonomous decisions, and executing tasks. Unlike traditional question-answering AI, agents possess the ability to call tools, perform multi-step reasoning, and act autonomously. Current mainstream AI programming assistants like GitHub Copilot, Cursor, and Devin essentially rely on large language models (LLMs) as their "brains," automating complex tasks through tool calls (such as code execution, file read/write, and web search). However, most agents' "memory" is limited to the context window of a single conversation. Once the task ends, the state resets to zero, and experience cannot accumulate across sessions—this is one of the core limitations of the current architecture.

An LLM's "memory" essentially depends on the attention mechanism of the Transformer architecture, and its effective information capacity is constrained by the number of tokens in the context window. GPT-4 supports around 128K tokens, and Claude 3 supports around 200K tokens—seemingly large, but far from enough for long-term projects. The source code of a medium-sized codebase can easily exceed this limit, let alone historical experience across sessions. This has given rise to the engineering need for "memory externalization": persisting an agent's important experiences in external systems (databases, files, knowledge graphs) and retrieving them on demand to inject into the context when needed, thereby breaking through the window limitation. This approach aligns closely with the theory of "External Memory" in human cognitive science—humans likewise externalize vast amounts of knowledge into books, notes, and databases rather than relying entirely on their brain's memory. The shared memory mechanism of the Crew project essentially elevates this externalization process from the individual level to the team level.

The next time it faces a similar problem, it often has to start from scratch. While this mode continuously strengthens individual capabilities, it has always lacked one critical dimension: collaboration and knowledge accumulation.

A developer named Onnokh recently shared a thought-provoking open-source prototype project on Reddit—Crew. It attempts to answer a simple yet profound question: What would happen if multiple AI agents within the same project could learn from each other and share their experiences?

Core Concept: A "Stack Overflow" Built for Agents

The author's positioning of Crew is quite vivid—"a Stack Overflow built by agents, for agents."

This metaphor precisely captures the essence of the project. Stack Overflow was founded in 2008, and its core value lies not just in Q&A itself, but in building a mechanism for the "structured accumulation of collective knowledge": vote-based ranking, accepted answer marking, tag classification, and full-text search have turned the problem-solving experience of millions of developers into a reusable, sustainable knowledge graph. As of 2024, Stack Overflow has accumulated over 58 million questions and answers, becoming the world's largest programmer knowledge base. The ingenuity of this mechanism lies in the fact that the value of knowledge is continuously validated and reinforced over time and through citation frequency, rather than being stacked linearly. In the world of human developers, the value of Stack Overflow doesn't lie in how brilliant any single developer is, but in the continuous accumulation of collective wisdom: the pitfalls one person has stumbled into and the solutions they've found are deposited in Q&A form, benefiting those who come after.

It's worth noting that Stack Overflow itself is also an excellent case study for researching "knowledge emergence"—the overall value of the platform far exceeds the simple sum of all its Q&A entries. This is because structured associations, cross-references, and the formation of community consensus give rise to nonlinear synergistic effects in the knowledge network. For Crew to truly realize this vision, it needs not only to "store experiences" but also to build a semantic association network of experiences between agents, allowing knowledge to produce chemical reactions across different tasks and different agents, rather than merely being retrieved as isolated entries.

Crew aims to transplant this mechanism onto AI agents:

• What one agent learns becomes a shared resource accessible to the entire team
• Agents are no longer isolated execution units, but collaborators who can "consult" each other's experiences
• As the project progresses, the team's overall knowledge base continuously grows

The author particularly emphasizes the goal—letting agent teams gradually "become better coworkers, not just better individual agents." This statement points out the fundamental difference between Crew and traditional AI programming tools.

Why "Team Collaboration" Is the Next Breakthrough

The Ceiling of Individual Optimization

Over the past two years, the progress of AI programming assistants has mainly focused on improving single-model capabilities: larger context windows, stronger code comprehension, and more accurate completions. But this path has an inherent bottleneck—no matter how powerful an individual is, if its experience cannot be passed on, the efficiency gains at the team level are quite limited.

Consider a typical development scenario: five agents are respectively responsible for the frontend, backend, testing, database, and deployment. The backend agent solves a tricky concurrency problem, but this experience cannot be perceived by the testing agent, so the team still repeatedly consumes resources on similar issues.

This limitation is known in systems complexity theory as the "local optimum trap"—each individual reaches the optimum on its own subproblem, but the overall system stays far from the global optimum due to information silos. In software engineering, this echoes the essence of Conway's Law: a system's architecture often mirrors the team's communication structure. If agents lack effective knowledge flow between them, the code architecture produced by their collaboration will also exhibit similar fragmentation. What Crew attempts to break through is precisely this structural barrier.

The Compound Effect of Shared Memory

The core idea of Crew is to build a layer of Shared Memory, which aligns closely with the recently hot topics of "agent memory" and "multi-agent orchestration" in the industry.

"Shared memory" in multi-agent systems is typically implemented through a vector database—converting an agent's experiences, solutions, or conversation summaries into high-dimensional vector embeddings, storing them in vector databases such as Pinecone, Weaviate, and Chroma, and making them available for other agents to retrieve and invoke through semantic similarity. This technical approach is also the core component of the current RAG (Retrieval-Augmented Generation) architecture.

An embedding is the process of converting discrete text, code, or experience into a numerical vector in a continuous high-dimensional space, so that semantically similar content is closer together in the vector space. Mainstream embedding models include OpenAI's text-embedding-ada-002, Google's Gecko, and others, typically with dimensions ranging from 768 to 3072. In code scenarios, embedding models need to specifically understand the syntactic structure and semantic equivalence of programming languages—for example, two pieces of code with identical functionality but different syntax should be close to each other in the semantic space. This poses a challenge for general-purpose embedding models and is precisely the core value of code-specific embedding models (such as GitHub's CodeBERT and Microsoft's UniXcoder). The core algorithm of vector retrieval is Approximate Nearest Neighbor (ANN) search, with common implementations including HNSW (Hierarchical Navigable Small World graphs) and IVF (Inverted File index), which can achieve millisecond-level retrieval responses at the scale of millions of vectors.

RAG was formally proposed by Meta AI in the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Its working principle is divided into two stages: in the offline stage, documents are split into chunks and converted into high-dimensional vectors via an embedding model to be stored in a database; in the online stage, the query is also vectorized and the most relevant chunks are retrieved through cosine similarity or the approximate nearest neighbor (ANN) algorithm, then concatenated into the prompt for the LLM to generate an answer. In Crew's multi-agent scenario, the challenge of the RAG mechanism lies in the complexity of the "write side"—the writing of a human knowledge base is gatekept by humans, whereas the writing of an agent knowledge base requires automated quality assessment, which is an open engineering problem without a standard answer yet.

Besides vector retrieval, the knowledge graph is another approach to structured shared memory, suitable for expressing complex relationships between entities. A knowledge graph organizes knowledge in the form of "entity-relationship-entity" triples, possessing precise logical reasoning capabilities, with representative implementations including graph databases such as Neo4j and Amazon Neptune. In agent scenarios, a knowledge graph can precisely express structured causal relationships such as "module A depends on module B" or "function X triggers a Z-type concurrency problem under the Y version environment"—relationships that fuzzy semantic vector retrieval struggles to capture precisely. However, building a knowledge graph itself requires higher structuring costs—automatically extracting high-quality triples from an agent's unstructured experience involves NLP techniques such as entity recognition and relation extraction, and its engineering complexity is significantly higher than that of vector retrieval. The two approaches each have trade-offs: vector retrieval excels at fuzzy semantic matching, while knowledge graphs excel at precise logical reasoning—which approach the Crew project chooses will directly determine the upper limit of its knowledge-sharing quality.

When knowledge can flow freely between agents, the growth of team capability exhibits a compound effect—each completed task not only achieves its immediate goal but also accumulates reusable experiential capital for all future tasks.

This is also the most noteworthy architectural intent of Crew: shifting the focus from "making a single agent smarter" to "making a group of agents smarter together."

Architectural Considerations and Open Questions

As an early prototype, Crew is currently more about throwing out concepts and seeking feedback. In the post, the author explicitly stated a desire for community input on both the concept itself and the architectural design. The project code is already open-sourced on GitHub (github.com/Onnokh/crew), and interested developers are welcome to join the discussion.

From an engineering perspective, several key questions still warrant deeper exploration for this vision to truly come to fruition:

Knowledge Quality and Filtering

Stack Overflow relies on voting and moderation mechanisms to ensure answer quality. If the experiences shared by agents lack filtering, a "contamination effect" where erroneous experiences are repeatedly cited is quite likely to emerge.

In machine learning, "Data Poisoning" is a thoroughly researched security threat—once malicious or erroneous data enters the training set, it systematically affects model output. In agent shared memory scenarios, similar risks appear in the form of "experience contamination": an erroneous solution formed by an agent in a special environment, if it enters the shared knowledge base without verification, may be incorrectly cited by other agents in different scenarios, forming cascading errors.

In software engineering, code quality assurance relies on multiple lines of defense such as test coverage, static analysis, and code review. Migrating this approach to an agent knowledge base, one can design similar multi-layer verification mechanisms: the first layer is "execution verification"—only solutions that pass automated tests are allowed to be written into the shared knowledge base; the second layer is "semantic deduplication"—using vector similarity to detect redundant or contradictory knowledge entries; the third layer is "confidence decay"—the credibility weight of knowledge entries gradually decays over time or as the environment changes, avoiding continued interference from outdated experience; the fourth layer is "citation verification"—tracking the task success rate after a piece of experience is cited, inferring knowledge quality from result feedback. The complexity of this mechanism is no less than that of designing a consistency protocol for a distributed database, and it is a core engineering challenge for moving multi-agent systems from prototype to production.

Human communities filter noise through peer review, reputation systems, and the test of time; AI systems may need to introduce mechanisms such as formal verification, automated test pass rates, or confidence scoring to establish a similar quality moat. How to evaluate and verify whether the knowledge an agent has acquired is reliable is a core engineering challenge.

Knowledge Representation and Retrieval

What exactly is shared between agents? Raw conversation logs, structured solutions, or abstracted "experience patterns"? Different representation forms directly determine whether knowledge can be effectively retrieved and reused.

This question corresponds in cognitive science to the classic topic of "Knowledge Encoding"—the tacit knowledge of human experts is often difficult to make explicit, which is precisely the root of the challenge of transmitting skills that "can only be sensed, not conveyed in words." An agent's "experience" may likewise have a similar distinction between the tacit and the explicit: process knowledge such as execution paths and tool-call sequences is easily recorded in a structured way, while the metacognitive judgment of "which solution to choose under what circumstances" is more difficult to formalize. If the Crew project can explore an effective "experience encoding language," it would provide an important reference for the entire AI Agent engineering field.

Conflict and Deduplication

When multiple agents give different solutions to the same problem, how does the team decide? This involves more complex coordination mechanisms such as version management and priority ranking, requiring further design investment.

This challenge has a deep theoretical foundation in the field of distributed systems. The CAP theorem states that a distributed system cannot simultaneously guarantee Consistency, Availability, and Partition tolerance. A multi-agent knowledge base faces a similar trilemma: knowledge consistency (all agents share the same "truth"), low-latency knowledge writing (not blocking task execution due to consensus mechanisms), and knowledge availability under network partitions—there are fundamental trade-offs among the three. Mature distributed consensus algorithms such as Raft and Paxos provide a systematic solution framework, but adapting them to the semantic knowledge level rather than the simple data value level remains an open research problem.

An Exploration Direction Worth Continued Attention

Crew is currently still a small prototype, some distance from being production-ready. But the approach it represents—shifting from optimizing individual agents to building multi-agent teams that can collaborate, share, and continuously evolve—is precisely one of the most imagination-inspiring exploration directions in the current AI Agent field.

Multi-Agent Systems are not a new concept. Their academic roots can be traced back to distributed artificial intelligence research in the 1980s—early representative work includes the Actor model proposed by Carl Hewitt (1973) and the Distributed Vehicle Monitoring Testbed (DVMT) led by Victor Lesser. In the 1990s, MAS gradually formed an independent discipline, and researchers began to systematically explore communication protocols between agents (such as the FIPA standard), negotiation mechanisms, and emergent behavior. However, limited by the computing power of the time and the immaturity of foundational AI models, MAS long remained at the academic level. It wasn't until after 2023 that the qualitative leap in LLM capabilities gave each agent genuine language understanding and reasoning abilities, and multi-agent systems truly ushered in an explosion of engineering practice.

It's worth mentioning that the FIPA (Foundation for Intelligent Physical Agents) protocol standard attempted in the late 1990s to establish a unified specification for multi-agent communication, defining the Agent Communication Language (ACL) and interaction protocols. Although the FIPA standard failed to be widely adopted due to its excessive complexity, its core idea—that agents need a shared "language" to negotiate, request, and share information—has been realized in a more natural way in today's LLM era: the large language model itself is a general-purpose semantic understanding and generation engine, naturally serving as the "Lingua Franca" between agents. This makes communication coordination in contemporary multi-agent systems far more elegant and resilient than in the FIPA era.

One of the most fascinating phenomena in multi-agent systems is "Emergent Behavior"—when a sufficient number of simple agents interact following local rules, complex capabilities can spontaneously emerge at the overall system level that no single agent could achieve alone. This phenomenon has rich counterparts in nature: ant colonies achieve complex path optimization through simple pheromone rules, and neurons produce consciousness and cognition through local synaptic connections. In AI multi-agent systems, the shared memory mechanism is precisely the key infrastructure for promoting emergence—it enables agents to form effective "pheromone trails," allowing collective wisdom to surpass the simple sum of individual capabilities. The true potential of Crew perhaps lies not only in "knowledge sharing" but in whether it can trigger cognitive emergence at the team level.

Currently, representative frameworks in the industry include: Microsoft's open-source AutoGen (supporting inter-agent conversation and collaboration), LangGraph launched by LangChain (a directed-graph-based multi-agent workflow), and CrewAI (focused on role-based agent team collaboration). It's worth noting that the Crew project and CrewAI are similar in name but differ in positioning—CrewAI focuses on task division and process orchestration, while Crew focuses more on the accumulation of experience and knowledge sharing across tasks; the two address different levels of the multi-agent collaboration problem.

As multi-agent systems gradually move from concept to engineering practice, "memory" and "collaboration" will become the key variables determining a team's upper limit. Although early attempts like Crew are still immature, they propose a clear value hypothesis: future AI development teams will no longer compete solely on whose model is the strongest, but on whose team is best at collaborating and most capable of accumulating knowledge.

For developers who follow AI Agent engineering practice, this open-source project is worth continued tracking, and worth contributing to personally, to jointly explore the boundaries of agent collaboration.