Hidden Markov Preference Models: Online Prediction of an Attacker's Next Move

In network attack and defense, defenders need to predict an attacker's next action before it happens, based on available information, and produce a probability distribution. This seems simple but is complicated by two structural complexities: an attacker's behavioral mode persists within the same scenario and may switch; and each action consists of an operation plus parameters of the state it depends on, with the set of feasible parameter values changing over time. A recent paper proposes using a hidden Markov model (HMM) to uniformly capture both aspects, and achieves performance superior to recurrent baselines on the CAGE 2 network defense benchmark.

Problem Background: The Dual Complexity of Attacker Behavior

Traditional attack prediction often assumes a fixed attacker behavioral mode, or ignores the dynamic feasibility of action parameters. In reality, however, attackers may switch strategies across different attack phases—for example, from reconnaissance to exploitation, or from lateral movement to data exfiltration. At the same time, the specific parameters of each action (e.g., "scan ports")—such as port numbers—are constrained by the current network state, and the set of feasible values changes over time. The paper points out that these two structures are intertwined: mode switching affects the distribution over action parameter choices, while parameter feasibility in turn provides clues for mode switching. Therefore, a probabilistic framework is needed that can simultaneously model mode switching and action parameter feasibility.

Hidden Markov Model (HMM) is a statistical Markov model used to describe observable sequences generated by an unobservable sequence of hidden states. It rests on two core assumptions: the hidden states form a first-order Markov chain, i.e., the current hidden state depends only on the previous hidden state; and observations depend only on the current hidden state. An HMM is fully determined by three sets of parameters: the initial state distribution, the state transition matrix, and the observation (emission) probabilities. Classic applications include speech recognition, biological sequence analysis, and handwriting recognition. In attack prediction scenarios, the attacker's behavioral modes (such as reconnaissance, exploitation, lateral movement) correspond to hidden states, while observable actions and their parameters correspond to observations. The HMM provides exactly the probabilistic framework for inferring modes from action sequences and predicting the next step.

Model Core: Hidden States and Feasibility Mask Trees

The proposed hidden Markov preference model treats the attacker's active behavioral mode as a latent state, and defines the action patterns under each mode on a "feasibility mask tree." Tree nodes correspond to executable actions, and masks dynamically block infeasible action branches according to the current state. In this way, the model can both characterize mode switching through hidden-state transitions and handle the time-varying feasibility of action parameters through the tree structure.

The key lies in the design of the likelihood function: the same likelihood is used both to compute the predictive distribution over the next action and to update mode beliefs after an action is observed. For partially observed action prefixes, the model contributes its already-resolved local factors, thereby supporting online real-time inference. This uniformity allows prediction and learning to connect seamlessly.

Learning and Theoretical Guarantees

For parameter estimation, the paper decomposes the problem into convex logistic regression fits; when mode labels are missing, the Baum-Welch algorithm can reuse these fitted results. This avoids the pitfalls of non-convex optimization and improves computational efficiency. Theoretically, the authors derive three guarantees:

• Arrival-weighted node-level KL expressions: quantify each node's contribution to mode information, reflecting differences in the information content of different action paths.
• Sequence-level log-loss bounds: provide a mathematical upper bound on prediction loss for mode-switching paths.
• Unified fallback guarantee: in extreme cases, model performance will not fall below a certain baseline.

These analyses provide a reliability basis for the model's practical application.

The Baum-Welch algorithm is a special case of the expectation-maximization (EM) algorithm for HMM parameter estimation, used to estimate transition and emission probabilities when hidden states are unobservable. It alternates between two steps: the E-step uses the forward-backward algorithm to compute the posterior distribution of hidden states at each time step given the current parameters, and the M-step updates the parameters accordingly to maximize the expected log-likelihood. The algorithm guarantees that the likelihood is monotonically non-decreasing, but it usually converges only to a local optimum, and its computational complexity grows with the square of the number of states. One contribution of this paper is to decompose the emission-parameter fitting in the M-step into convex logistic regression problems, thereby avoiding the local-optimum risk of non-convex optimization and allowing these convex fitting results to be reused when mode labels are missing.

Benchmark Performance

On the CAGE 2 network defense benchmark, the paper reports quantitative results: the filtered hidden states are worth 0.8 nats per step, meaning the hidden states carry significant uncertainty-reduction information. More strikingly, after a controller switch, the model's loss begins to decrease within 3 steps, whereas the recurrent baseline requires 10 steps. This shows that the HMM can adapt more quickly to abrupt changes in an attacker's behavioral mode, offering a practical advantage in dynamic adversarial environments.

According to the authors' analysis, this advantage stems from the model's explicit modeling of mode switching—when the attacker changes strategy, the hidden-state belief updates rapidly, thereby guiding adjustments in the predictive distribution; the recurrent baseline, by contrast, relies on the slow evolution of hidden-layer states and responds with a lag.

CAGE 2 is a public benchmark environment for network defense, short for Cyber Attack Generator Environment 2, driven by the CybORG simulation framework and used to evaluate autonomous cyber defense agents. It simulates a small enterprise network with multiple hosts and subnets. The red agent (attacker) can perform scanning, exploitation, privilege escalation, lateral movement, and other actions, while the blue agent (defender) must detect and respond. CAGE 2 provides standardized observation interfaces and reward signals and supports multiple red strategies (such as limited observability, fixed strategy, and random strategy), so it is often used as a comparison platform for attack prediction and defense decision research. This paper compares the HMM with recurrent neural network baselines on this benchmark, using per-step predictive log-loss as the metric.

Implications for Security Defense

This research provides a structured path for online attack prediction: using hidden states to capture behavioral modes, a feasibility tree to handle action parameters, and a unified likelihood to enable online learning. For defenders, this means earlier warning of an attacker's next move, especially at critical points when the attacker switches strategy. However, practical deployment of the model still needs to consider computational overhead and the cost of constructing the tree structure. Future work could explore how to incorporate domain knowledge into mask design and extend to multi-attacker scenarios.

Overall, this paper strikes a balance between theoretical rigor and empirical effectiveness, offering new ideas for sequential decision-making in adversarial environments.

One-line Summary

Using a hidden Markov model to uniformly capture attacker mode switching and time-varying feasibility of action parameters, outperforming recurrent baselines on CAGE 2.

Paragraph Summary

The article introduces a study on cyber attack prediction: an attacker's behavioral mode persists within the same scenario and may switch, while the feasible values of action parameters change with network state—and these two structures are intertwined. The paper proposes a hidden Markov preference model that treats behavioral modes as hidden states, defines action patterns on a feasibility mask tree to dynamically block infeasible parameters, and uses a unified likelihood to simultaneously predict the next action and update mode beliefs, supporting online inference under partial observation. Parameter estimation is decomposed into convex logistic regression fits that Baum-Welch can reuse, avoiding non-convex optimization. The theoretical part provides node-level KL expressions, sequence-level log-loss bounds, and a unified fallback guarantee. On the CAGE 2 benchmark, the filtered hidden states are worth 0.8 nats per step, and after a controller switch the loss begins to decrease within 3 steps, whereas the recurrent baseline needs 10—showing faster adaptation to mode shifts.