BETS

BETS (Belief Embedding Tree Search) is a Monte Carlo planner that satisfies three key requirements: the belief representation can be computed from the parent node via POMDP dynamics, the search can sample from it, and it can be learned to provide the information needed for the policy and value functions.

BETS uses a generative distribution embedding (GDE) to represent the belief at each node and updates it through the POMDP dynamics.

BETS jointly optimizes the belief model (GDE), policy network, and value network during training, and the belief model's loss minimizes the discrepancy between the generated distribution and the true hidden state, commonly using negative log-likelihood or maximum mean discrepancy.

Because the belief representation is differentiable, the gradients of the policy and value networks can be propagated back to the belief model, enabling end-to-end co-adaptation.

The study evaluated BETS on six configurations across three classic POMDP environments, RockSample, LightDark, and PocMan, with baselines including BetaZero and three planners without learning components.

BETS is either competitive or leading on all six configurations, while each baseline lags far behind on at least one configuration.

In the configuration with the highest belief dimension, BETS substantially surpasses all compared methods, and even maintains its advantage without performing search.

BETS relies on the accessibility of the POMDP dynamics to update the belief embedding, and how it applies in scenarios where the dynamics are unknown or difficult to model remains to be explored.

The training objective of BETS's belief model requires the environment to provide the true hidden state as a supervision signal, which may not be available in some practical applications.