When Q-learning Hyperparameters Are Dictated by Personality: A Reinforcement Learning Design Retrospective of a Civilization Simulator

When Every NPC Has Its Own Learning Rate and Discount Factor

In reinforcement learning textbooks, alpha (learning rate) and gamma (discount factor) are usually global hyperparameters—tune them once and leave them fixed. But a developer on Reddit shared his approach in a civilization simulator: each individual is an independent tabular Q-learner, and their hyperparameters are derived directly from personality traits.

The mapping is as follows:

• alpha = 0.08 + memory_strength × 0.14, ranging from 0.08 to 0.22
• gamma = 0.82 + curiosity × 0.12, ranging from 0.82 to 0.94
• epsilon decays with age: 0.30 − age × 0.00005, floor 0.08

This design is compelling at the narrative level—agents with good memory learn fast, and curious agents value long-term rewards more. But from a control-theory perspective, it hands the two most sensitive hyperparameters over to a personality distribution that evolves over time, making stability issues almost inevitable.

Pitfalls of Personality-Driven Hyperparameters

An upper bound of 0.22 for the learning rate is on the high side for tabular Q-learning. With single-step updates, new experience overwrites a substantial proportion of the historical estimate; if the reward signal itself is noisy, high-alpha agents are prone to Q-value oscillation, manifesting as erratic flip-flopping behavior. The original author explicitly asked whether "trait-driven alpha and gamma are asking for stability trouble," and that concern is well-founded.

A gamma near 0.94 is also uncomfortable. The higher the discount factor, the longer the variance propagation chain in the bootstrap target, and combined with high alpha, errors get amplified repeatedly. More troublesome is that these two are controlled independently by different traits, meaning a "high alpha + high gamma" combination can occur—a configuration that traditional tuning would go to great lengths to avoid.

A compromise approach is to couple alpha and gamma negatively, or to impose an alpha cap that decreases with update count, enabling fast early exploration and stable late convergence.

The tabular Q-learning update rule is Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)]. α controls the proportion of each new experience that overwrites the old estimate, while γ determines the weight of future rewards in the current decision. Together they affect the variance of the bootstrap target: the larger α is, the more noise each update introduces; the closer γ is to 1, the farther value-estimation variance propagates along the time chain. Classic tuning wisdom holds that α and γ trade off against each other—high γ requires smaller α to suppress variance accumulation, and vice versa. Binding each to an independent personality trait abandons this coupling constraint, allowing the system to sample parameter combinations like "high α + high γ" that would be ruled out outright in manual tuning. This is the root cause of the original author's stability concerns.

A Known Flaw in Reward Shaping

The original poster candidly admitted a bug: "the wart."

Novelty Reward and Fear/Resilience Scaling

He layered three forms of shaping on top of the raw reward:

1. Novelty reward: when an agent tries an action in a state for the first time and the reward is non-negative, it receives a bonus of 0.004 + curiosity × 0.008
2. Negative reward scaling: negative outcomes are amplified by fear by about 1.45× and attenuated by resilience by up to 20%, so the same bad outcome hurts timid agents more
3. Inactivity penalty: any reward below 0.006 gets 0.006 subtracted

The logic of the first two is self-consistent—fear amplifies negative experiences, resilience buffers blows—matching the personality setup. The problem lies in the threshold of the third.

16% of Updates Went Negative

That 0.006 inactivity penalty inadvertently hits small positive rewards. The author's own tally: about 16% of update outcomes are negative, of which 98% are actually "repeated successful actions" that got pushed negative by the penalty. In other words, the agent did the right thing and received small positive feedback, but the system judged it "not active enough" and deducted points.

This creates false negative associations with reliably effective behaviors, and over time suppresses exploitation—precisely the behavior pattern most needed once Q-learning converges. The author's proposed fix is to explicitly pass a success flag so the update logic can distinguish "genuinely did nothing" from "did something but the payoff was small." He also recognizes that this will change learning trajectories across all worlds, so a seed sweep to compare before and after is needed. This attitude deserves more credit than a direct patch.

Reward shaping refers to artificially adding auxiliary reward terms on top of the environment's raw reward to guide the agent toward desired behavior faster. Its theoretical basis is the potential-function method proposed by Ng et al.: if the shaping term can be expressed as the difference of some potential function, the optimal policy is unchanged; otherwise it may introduce policy bias. The novelty reward and inactivity penalty here are both non-potential-based shaping, so they change the objective the agent actually optimizes. The fixed-threshold penalty in particular conflates "did something but the payoff was small" with "did nothing at all," creating a misalignment between the reward signal and the true task objective. The author's success flag is essentially introducing a binary event indicator into the shaping term, thereby decoupling "whether a valid interaction occurred" from "how large the interaction payoff was."

Bootstrap Masking: Correctness Hides in the Details

The original post mentions that the bootstrap target is masked: taking the max Q over actions actually available in the next state. If the next state has no corresponding Q-table row yet, the max is taken as 0 and the mask computation is skipped.

This branch handling merits a couple of remarks. In tabular Q-learning, initializing unseen states to 0 is common practice, but it introduces optimistic bias—especially when all rewards are positive, where 0 is actually a pessimistic estimate. A more rigorous approach is to initialize with the mean or minimum Q of the state's seen actions. However, for a simulator scenario where the state space explodes and most states are visited only a few times, 0-initialization is engineering-acceptable, as long as one is aware that it biases early exploration toward unvisited regions.

Another detail is that the mask depends on the triple "era, place, tribe knows," indicating that action availability itself is dynamic. This increases the frequency of mask computation, but also brings the bootstrap target closer to the real decision space, avoiding the classic error of taking max over illegal actions.

Bootstrap is the core mechanism of temporal-difference learning: updating the current state's value using the current estimate of the next state's value, without waiting until the end of an episode. In tabular Q-learning, if the next state s' has never been visited, its Q-table row does not yet exist, and the max operation cannot be computed. The common handling is to initialize all action values of new states to 0. When all rewards are positive, 0 is low relative to the true expected return, so the agent tends to treat unvisited states as "not worth going to," producing pessimistic bias; conversely, if rewards include negative values, 0 may be overly optimistic. More robust approaches include initializing with the mean of seen actions, initializing with the global average return, or introducing visit-count-based optimistic initialization (as in R-max-style algorithms). In this article's scenario, the state space keeps expanding as the civilization evolves and most states are visited only a handful of times, so 0-initialization is engineering-acceptable, but one must be aware that it affects the direction of early exploration.

Several Directions Worth Trying

Overall, the most valuable part of this design is binding personality to learning dynamics—and that is also its most dangerous part. If I were to offer suggestions:

Fix the inactivity penalty first. This is a clear logic error with a quantifiable impact (98% of the 16% negative updates are false hits). Add the success flag first, then run a seed sweep to confirm the fix doesn't break other emergent behaviors.

Add a cap or decay to alpha. 0.22 is acceptable as an initial value, but it should decay with the visit count of that state-action pair, e.g., divided by the square root of the visit count. This preserves personality differences while avoiding long-term oscillation.

Monitor the behavior of high-alpha, high-gamma combinations. You can track the Q-value variance and action-switching frequency of such agents; if they are significantly above the population mean, the stability problem is real and constraints should be added to the trait mapping.

Consider merging the novelty reward and the inactivity penalty. Both fundamentally encourage exploration, but one adds and the other subtracts, so they logically conflict. Unifying them into a single exploration bonus that decays with visit count would be cleaner.

The code is in crates/sim-core/src/organism/organism/learning.rs; the project is open-sourced under the MIT license, and the author also provides a demo for interactively observing the learning process. For developers interested in multi-agent reinforcement learning or social simulation, this is a case worth reading against your own implementation—especially the failure modes it exposes, which are more instructive than success stories.

One-line Summary

A civilization simulator drives each NPC's Q-learning hyperparameters with personality traits—stability and reward-shaping flaws are worth watching out for.

Paragraph Summary

The article analyzes a civilization simulator project: each NPC is an independent tabular Q-learner whose learning rate alpha and discount factor gamma are mapped from memory_strength and curiosity, with epsilon decaying with age. This personality-bound hyperparameter design is narratively compelling, but from a control-theory perspective it harbors stability risks—the alpha cap of 0.22 is high, gamma can reach 0.94, and the two are controlled independently by different traits, allowing dangerous high-alpha-plus-high-gamma combinations. On reward shaping, the 0.006 inactivity penalty threshold inadvertently hits small positive rewards, causing about 16% of updates to go negative, 98% of which are repeated successful actions wrongly penalized. Bootstrap masking takes 0 for unseen states and skips computation, which is engineering-acceptable but biased. The author proposes fixing it with a success flag, and suggests decaying alpha with visit count, monitoring high-alpha high-gamma combinations, and merging exploration-related reward terms.