Populace
An open-source AI agent evaluation framework that generates a simulated town of 200 AI residents, each with memory and social networks, to conduct multi-turn, longitudinal stress tests on agents under evaluation. It reveals failure patterns that single-turn dialogue benchmarks cannot detect.
Timeline (last 90 days)
Populace is an open-source project that generates 200 AI residents—each with a residence, job, family, finances, and an acquaintance network—from a single descriptive sentence, serving as a simulated town environment for testing other agents.
Populace was built by an 18-year-old developer as their first open-source project, uses the MIT open-source license, and is hosted on GitHub (populace-sim/populace).
In Populace's controlled experiment, an LLM customer service agent (Qwen3-32B) that could view customer accounts and a keyword-matching bot were tested in the same town, with the same random seed, and with the same questions.
In the experiment, the LLM customer service agent resolved 7 issues on the first day, while the keyword-matching bot resolved only 4.
In the experiment, the LLM customer service agent broke 6 out of 8 due commitments, while the keyword-matching bot broke only 2 out of 30 commitments.
In the experiment, the LLM customer service agent told six different customers that an engineer would arrive at 10 a.m., while those time slots were actually booked for other households.
Populace's underlying architecture draws on the Generative Agents research paradigm, which originates from the paper 《Generative Agents: Interactive Simulacra of Human Behavior》published by Stanford in 2023.
In Populace, residents require approximately 32B-level models to behave convincingly; a 7B model never once contacted customer service.
On a single RTX 4090 GPU, each in-game day in Populace takes approximately 10 minutes, and the system also provides a free mock mode that requires no model.
Populace reports track resident errors and agent errors separately, in order to distinguish user noise from agent mistakes.
1 more timeline events
All Facts (11)
An agent integrated with Populace can be any Python object with a handle(message, ctx) method or any HTTP endpoint.
50%UnverifiedPopulace was built by an 18-year-old developer as their first open-source project, uses the MIT open-source license, and is hosted on GitHub (populace-sim/populace).
50%UnverifiedPopulace is an open-source project that generates 200 AI residents—each with a residence, job, family, finances, and an acquaintance network—from a single descriptive sentence, serving as a simulated town environment for testing other agents.
50%UnverifiedIn the experiment, the LLM customer service agent resolved 7 issues on the first day, while the keyword-matching bot resolved only 4.
50%UnverifiedIn the experiment, the LLM customer service agent broke 6 out of 8 due commitments, while the keyword-matching bot broke only 2 out of 30 commitments.
50%UnverifiedIn the experiment, the LLM customer service agent told six different customers that an engineer would arrive at 10 a.m., while those time slots were actually booked for other households.
50%UnverifiedPopulace's underlying architecture draws on the Generative Agents research paradigm, which originates from the paper 《Generative Agents: Interactive Simulacra of Human Behavior》published by Stanford in 2023.
50%UnverifiedIn Populace, residents require approximately 32B-level models to behave convincingly; a 7B model never once contacted customer service.
50%UnverifiedOn a single RTX 4090 GPU, each in-game day in Populace takes approximately 10 minutes, and the system also provides a free mock mode that requires no model.
50%UnverifiedPopulace reports track resident errors and agent errors separately, in order to distinguish user noise from agent mistakes.
50%UnverifiedIn Populace's controlled experiment, an LLM customer service agent (Qwen3-32B) that could view customer accounts and a keyword-matching bot were tested in the same town, with the same random seed, and with the same questions.
50%