[KongchangAI]
· 2 min read· 1,434 words

ScenarioForge: Making LLM-Generated Cyber Ranges Truly Deployable and Verifiable

ScenarioForge: Making LLM-Generated Cyber Ranges Truly Deployable and Verifiable

ScenarioForge binds LLM-generated security exercise plans to real resources, deploying and validating them in CORE for step-level verifiability.

ScenarioForge addresses the engineering gap where LLM-generated cyber range plans "read plausibly but won't run." Via a three-layer workflow—requirements parsing, generation, execution—it converts high-level requirements into resource-grounded topologies, services, traffic, segmentation, and challenge chains, instantiated and strictly runtime-validated in the CORE emulator. Its dataset includes 226 materialized scenarios, 294 Vulhub images, and 85 resource generators, all passing six build and validation gates and decomposed into 867 independently verifiable challenge objectives. In a 50-prompt benchmark, ScenarioForge with Qwen3.8 and GPT5.6 both achieved 100% topology construction and intent coverage. This work offers an infrastructure-level contribution to repeatable, verifiable security training and evaluation, though applicability is limited by the CORE and Vulhub ecosystems.

Cyber ranges and CTF (Capture The Flag) challenge construction have long been heavy manual labor for security teams: authors must simultaneously coordinate infrastructure, vulnerable services, attack paths, artifacts, and success conditions. Large language models can write seemingly detailed exercise plans, but there is a massive engineering gap between "reads plausibly" and "actually runs." ScenarioForge attempts to bridge this gap—it transforms high-level requirements into resource-grounded topologies, services, traffic, segmentation, and challenge chains, and completes instantiation and verification in the Common Open Research Emulator (CORE).

ScenarioForge paper source screenshot

From "Textual Plans" to "Executable Environments"

The core claim of ScenarioForge is that LLM-generated cyber exercises must be resource-grounded, not merely停留在 descriptive level. Resource grounding means every generated service, image, segmentation rule, and challenge objective is bound to a real, existing resource rather than a configuration the model fabricated out of thin air.

The system workflow can be summarized in three layers: the requirements-parsing layer decomposes high-level requirements into topology and resource constraints; the generation layer outputs services, traffic patterns, network segmentation, and challenge chains; the execution layer completes deployment in the CORE emulator and performs strict runtime validation and artifact checks. This pipeline aligns "generation" directly with "executability," skipping the manual step of translating plans and then debugging them.

CORE (Common Open Research Emulator) is an open-source network emulator primarily used to reproduce complex network topologies on a single physical machine. It creates isolated virtual nodes through lightweight virtualization technologies (such as Linux network namespaces) and connects these nodes with virtual links, thereby simulating routing, switching, and traffic behavior in real networks. Compared with solutions based entirely on virtual machines or container orchestration, CORE has low startup overhead and flexible topology configuration, making it suitable for scenarios that require repeatedly building and tearing down exercise environments.

In ScenarioForge, CORE plays a key role in the execution layer: the generated topologies, service deployments, and segmentation rules must ultimately be instantiated as operable virtual networks in CORE and subjected to runtime validation. This means the system's "resource grounding" claim depends on CORE's ability to simulate real network behavior—verifying whether a challenge objective is achieved essentially means checking, in the network emulated by CORE, whether the attack path works and whether artifacts appear as expected. Therefore, CORE's emulation fidelity and the types of images it supports also directly constitute the current capability boundaries of ScenarioForge.

226 Scenarios and 867 Verifiable Steps

The scale of the ScenarioForge Execution Dataset presented in the paper is noteworthy: the dataset contains 226 materialized scenarios, covering 294 Vulhub images and all 85 self-developed challenge resource generators (these generators were developed with AI-assisted scaffolding).

Crucially, all 226 scenarios passed six gates: build, preview, ordering, deployment, strict runtime validation, and live artifact inspection. They are decomposed into 867 steps, each of which is an independently verifiable challenge objective, with 457 supported by vulnerability instances and 410 supported by generated-resource instances. This "step-level verifiability" design is what distinguishes it from ordinary LLM-generated plans: security teams can directly see whether each objective is actually achieved, rather than accepting an entire plan as a package.

Vulhub is a widely used open-source collection of vulnerability ranges, providing reproducible environments for a large number of real-world vulnerabilities (such as classic vulnerabilities in Struts2, Log4j, Fastjson, etc.) in the form of Docker images. Each image typically encapsulates a specific version of an application and its dependencies, and users can directly launch a service instance with a known vulnerability after pulling it, for security research, vulnerability reproduction, and training.

In the ScenarioForge dataset, the 294 Vulhub images constitute the main resource source for the "vulnerability-supported instances" step category. Their value lies in the fact that these images have already been verified by the community as runnable and the vulnerabilities genuinely exist, effectively providing a reliable resource foundation for the LLM generation pipeline. The generation layer does not need to construct vulnerable environments from scratch but instead selects from these validated images and orchestrates them into the topology, thereby reducing the risk of "reads plausibly but won't run." However, this also means that the vulnerability types and version ranges covered by the dataset are limited by what Vulhub includes, and generalization capability across image sources or self-developed vulnerability environments remains to be verified.

Benchmark: Scaffolding Enables Frontier Models to Pass All Tests

In an independent benchmark of 50 matched prompts, ScenarioForge with Qwen3.8 and with GPT5.6 both achieved 50/50 (100%) topology/build success rates and 160/160 intent field coverage. By contrast, direct frontier generation only reached 41/50 build success and 147/160 intent fields after incorporating the full cumulative scaffolding (four types of guidance: structure, vocabulary, policy, and container image guidance).

This comparison needs careful interpretation: it shows that general models have obvious gaps in grounded topology and intent parsing when there is no structured guidance at all; and ScenarioForge's scaffolding mechanism (injecting domain knowledge into the generation pipeline) is the main source that fills this gap. It should be clear that this test does not cover ScenarioForge's downstream challenge resource generation, dependency/flag ordering, deployment, and live validation—that is, the benchmark only measures the first half, and the full capability of the actual system lies outside the test.

Scaffolding in the LLM context refers to structured guidance information attached before and after model calls, used to constrain and complete the model's output space. ScenarioForge's cumulative scaffolding includes four types: structural guidance (specifying the topology and configuration format the output should follow), vocabulary guidance (limiting the available range of domain terms and resource names), policy guidance (specifying compliance rules for segmentation, traffic, and challenge chains), and container image guidance (providing a list of available images and their usage).

These guidance mechanisms act like "domain knowledge injection": the model no longer improvises freely but makes choices and combinations within a given resource vocabulary and format constraints. In the benchmark, direct frontier generation had notably insufficient build success rates without scaffolding, and only reached 41/50 after incorporating the full cumulative scaffolding, indicating that the main shortcoming of general models lies in translating security domain constraints into grounded configurations rather than in natural language description ability itself. ScenarioForge's contribution can therefore be understood as engineering and systematizing this scaffolding and connecting it with the downstream execution and validation pipeline.

Implications for Security Training and Evaluation

ScenarioForge's value goes beyond saving time on building ranges. Repeatable, verifiable exercise environments are an infrastructure-level contribution to both security training and model evaluation. Training scenarios can be generated on demand and automatically validated, and evaluation results thereby become reproducible—the same challenge objectives can still be grounded into executable configurations on another generation run.

The limitations are equally obvious: binding to the CORE emulator, the Vulhub image ecosystem, and the coverage of self-developed resource generators determine the system's current scope of applicability. Generalization capability across emulators and image sources is not answered by the paper. However, the idea of "resource grounding + step-level validation" provides a clear engineering paradigm for LLM-generated deployable security environments.

Frequently Asked Questions

What is the difference between ScenarioForge and directly asking a large model to write a range plan?

Direct generation only produces textual descriptions and cannot guarantee that configurations actually exist or are deployable; ScenarioForge binds outputs to concrete resources and completes deployment and runtime validation in CORE.

What are the 867 steps in the dataset?

Each step is an independently verifiable challenge objective, including two categories: vulnerability-supported instances and generated-resource-supported instances, all of which pass live artifact inspection.

What do the benchmark results show?

They measure topology construction and intent field filling capability, showing that structured scaffolding significantly narrows the gap in direct generation by general models, but they do not cover downstream resource generation and deployment validation.

One-line Summary

ScenarioForge binds LLM-generated security exercise plans to real resources, deploying and validating them in CORE to achieve step-level verifiability.

Paragraph Summary

ScenarioForge attempts to solve the engineering gap where LLM-generated cyber range plans "read plausibly but won't run." Through a three-layer workflow of requirements parsing, generation, and execution, it transforms high-level requirements into resource-grounded topologies, services, traffic, segmentation, and challenge chains, and instantiates and strictly runtime-validates them in the CORE emulator. Its dataset contains 226 materialized scenarios, 294 Vulhub images, and 85 resource generators, all passing six build and validation gates and decomposed into 867 independently verifiable challenge objectives. In a 50-prompt benchmark, ScenarioForge with Qwen3.8 and GPT5.6 both achieved 100% topology construction and intent coverage, while direct frontier generation only approached this after full cumulative scaffolding. This work provides an infrastructure-level contribution to repeatable, verifiable security training and evaluation, but its applicability is limited by the CORE and Vulhub ecosystems.

#ScenarioForge#CORE emulator#LLM-generated cyber ranges#Vulhub#security training#runtime validation#topology construction
Share:

Related articles