Every Sim Is a Q-Learner: Offspring Inherit Mom's Q-Table and Mutate It

An Unconventional Take on a Civilization Sandbox: Making Every Person a Tabular Q-Learner

In reinforcement learning (RL), multi-agent simulations usually have every agent share a single policy network, or use centralized training with decentralized execution. But a side project shared by a developer on Reddit took a different path: in a civilization sandbox, every simulated person is an independent tabular Q-learner, and newborns inherit their mother's Q-table along with a small mutation. The project is called "thehumanbox," written in Rust and compiled to WebAssembly, running entirely in the browser.

According to the developer, the sandbox is fairly large in scale: each simulated person's state space is represented by hand-built short strings, covering perceptual information such as hunger, thirst, the nearest food and water, nearby kin, shelter, and danger. The action space has roughly 5000 possible actions, limited by era, age, location, and the technologies the tribe has discovered. Action selection prioritizes survival needs first, then follows an ε-greedy policy based on learned values.

Inheritance Mechanism: "Family Drift" of the Q-Table

The most compelling design in this project is how learning is passed across generations. When a baby is born, it receives a copy of its mother's Q-table, with about 20% of the values nudged (±0.03). This means that beyond an individual's learning over its lifetime, behavior also drifts along family lineages. In addition, a child's traits are blended and mutated from both parents, and the child also inherits the mother's vocabulary and her attitudes toward other tribes.

The developer openly admits that this design is meant to explore the feasibility of "learned results as a heritable unit." From an RL perspective, this amounts to treating the Q-table as a kind of "behavioral genome," letting evolutionary pressure and individual learning intertwine on the same timescale.

ε-greedy policy is the most common exploration-exploitation trade-off strategy in reinforcement learning. The agent picks a random action with probability ε (exploration) and picks the action with the highest current estimated Q-value with probability 1-ε (exploitation). ε typically decays over time, shifting the agent from heavy early exploration toward exploiting learned knowledge. In this project, action selection also layers on survival-need priority: when metrics like hunger or thirst exceed a threshold, the corresponding action is executed first, and ε-greedy selection is applied among the remaining actions. This "rules + learning" hybrid design is common in complex sandboxes, because pure ε-greedy is extremely inefficient at random exploration when the action space is around 5000, and hand-written priorities can greatly accelerate early learning.

Known Weaknesses: Hand-Built States and Reward Shaping

The developer is well aware of the limitations of this setup and actively seeks feedback. The two main weaknesses are: the state space is hand-built strings, and the reward function is hand-shaped. Both are sensitive topics in traditional RL research—hand-crafted state representations are hard to generalize, and hand-crafted rewards can easily introduce bias or lead to unintended behavior. However, for complex systems like a civilization sandbox, fully end-to-end learning is still unrealistic, so hand design is often a necessary compromise.

Reward shaping refers to artificially adding auxiliary reward signals on top of the original sparse reward to guide the agent toward desired behavior faster. For example, in a civilization sandbox, beyond the ultimate goal of "survival," small rewards might be given for subgoals like drinking water, eating, or staying near shelter. Shaping can significantly alleviate learning difficulties under sparse rewards, but classical theory (such as the 1999 work by Ng et al.) points out that inappropriate shaping changes the optimal policy, causing the agent to chase auxiliary rewards rather than the true goal—i.e., "reward hacking." The developer's remark that "hand-crafted rewards easily introduce bias or lead to unintended behavior" refers precisely to this risk. In open-ended environments like civilization simulations, where defining a single scalar objective is difficult, how to design shaping terms is itself a core challenge.

Technical Implementation: A Browser Simulation with Rust + WASM

The project is written in Rust and compiled to WebAssembly, allowing the entire civilization simulation to run in the browser without a backend. The relevant code lives in the open-source repository (MIT license) at crates/sim-core/src/sim/agents/growth.rs (inheritance logic) and organism/organism/learning.rs (Q-table updates). The developer also provides an online playable demo so anyone interested can try it directly.

WebAssembly (WASM) is a portable binary instruction format that modern browsers can execute efficiently. After Rust is compiled to WASM, compute-intensive logic can run in the browser at near-native speed while interacting with the page through JavaScript glue code. For this project, that means the entire multi-agent simulation—including updates and action selection for thousands of Q-tables—happens client-side, with no server compute required and anyone able to open a web page and play. The cost is that data passing between the WASM module and JS requires serialization, and the debugging experience is worse than native. This "Rust + WASM" combination has become increasingly popular in recent years for side projects in games, simulations, and edge computing.

A Few Points Worth Discussing in the RL Community

Although this is a side project, it touches on several interesting questions: does tabular Q-learning still have vitality in highly complex, non-stationary multi-agent environments? Can Q-table inheritance plus mutation effectively accelerate adaptation? How should hand-crafted rewards and hand-crafted states be balanced in civilization simulations? The developer explicitly says he is curious about the RL community's views, especially on state-space and reward design.

For people who enjoy multi-agent simulations and evolutionary computation, this project offers a runnable, modifiable sandbox that may spark more discussion about "combining learning and evolution."

Tabular Q-learner is the most basic class of value-function methods in reinforcement learning. It uses an explicit lookup table (usually a dictionary or 2D array) to store the Q-value of each "state-action" pair, i.e., the estimated long-term return. At each step, the agent looks up the row corresponding to the current state, picks the action with the highest Q-value (or explores via an ε-greedy policy), and then updates the Q-value of the chosen action using the temporal-difference (TD) error:

$$Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a'} Q(s',a') - Q(s,a)]$$

where $\alpha$ is the learning rate and $\gamma$ is the discount factor. Unlike deep Q-networks (DQN), which use neural networks to generalize across states, tabular methods memorize each state separately and therefore have no generalization ability—but they are interpretable, simple to update, and naturally suited to scenarios where "the table itself is a copyable object." In this project, the Q-table is the entirety of each simulated person's behavioral knowledge; inheriting a Q-table means inheriting a complete state-action value mapping.