PAWS Dataset: Policy-driven Agentic World Simulation for Finance

Making Policy Simulation Accountable

Policy interventions are never isolated decisions — they cascade through public communication, institutional responses, and stakeholder reactions in layered waves. Yet in financial multi-agent simulation, a persistent gap has remained: existing datasets rarely tie these propagation processes to time-aligned historical evidence. Researchers have largely depended on synthetic data or simulation environments with no traceable sources, making it difficult to validate results.

PAWS (Policy-driven Agentic World Simulation) is designed precisely to address this gap. It establishes an auditable, historically grounded data foundation for evaluating agent influence, policy response cascades, and the alignment between behaviors and outcomes.

Dataset Scale and Structure

PAWS covers 36 verified U.S. financial and economic policy events, linked to 12,727 policy-related news records and 65,291 source-backed stakeholder actions. One of its defining characteristics is that every action is linked to the supporting news evidence that grounds it — nothing is generated from thin air.

Equally noteworthy is how actions are structurally represented. Each action is described through a multi-layer event frame that captures four dimensions:

• Interaction mode: how the acting entities engage with one another
• Financial behavior family and subtype: categorical classification of the action
• Semantic attributes: specific meaning characteristics of the action
• Condition mapping: correspondence to external taxonomies

In addition, the dataset normalizes entity resolution by mapping diverse actors to standardized organizational entities, and aligns actions with daily market return context — enabling genuine policy-agent simulation replay.

Annotation Quality: Dual Validation by AI and Humans

For a dataset claiming to be auditable, annotation quality is central to its credibility. PAWS was validated on a stratified sample of 2,522 actions: independent AI reviewers and human reviewers reached an initial agreement rate of 89.4% on interaction mode classifications, with disagreements resolved through an arbitration process.

This dual-track validation framework leverages the scalability of AI annotation while maintaining human judgment as a quality backstop, providing empirical support for the dataset's reliability. An initial agreement rate approaching 90% is considered high for annotation tasks requiring this level of semantic understanding.

Two Case Studies: Reconstructing Policy Timelines from Historical Events

PAWS validates its reconstruction capability through two landmark financial events.

The first is the 2008 short-selling ban, a major regulatory intervention during the financial crisis. The second is the 2001 decimalization reform, the market structure change that converted U.S. stock quoting from fractions to decimals.

The research shows that PAWS can reconstruct well-documented policy timelines and corresponding market patterns for both events — and critically, under both news-dense and news-sparse conditions. This last point matters significantly, because media coverage of real-world policy events is rarely uniform, and the ability to reconstruct under sparse conditions reflects the dataset's robustness.

The 2008 short-selling ban was an emergency order issued by the SEC in September 2008, explicitly prohibiting naked short-selling on approximately 800 financial stocks for roughly three weeks. It was enacted against the backdrop of the panic triggered by Lehman Brothers' collapse, with regulators attempting to prevent short-selling from accelerating the stock price collapse. However, academic research on the ban's actual effectiveness is highly contested — multiple studies found that it not only failed to stabilize prices but also reduced market liquidity and widened bid-ask spreads. This makes it a classic case study of "regulatory intent diverging from market outcome," and an ideal test for whether a policy simulation system can reconstruct complex feedback mechanisms.

The 2001 decimalization reform was a structural change in which major U.S. exchanges uniformly switched the minimum price increment from 1/16 of a dollar (approximately 6.25 cents) to $0.01, aiming to align with international markets and lower retail trading costs. The direct consequence was a sharp narrowing of bid-ask spreads, but it also catalyzed the rise of high-frequency trading strategies and profoundly altered market microstructure. These two events span different policy types (crisis response vs. structural reform) and differ significantly in news coverage density and market impact pathways, forming a demanding stress test of PAWS's reconstruction capability.

The Hidden Risk Behind High Accuracy: Detecting Rare Actions

A replay study using PAWS revealed an easily overlooked problem: high overall accuracy can mask the failure to detect rare stakeholder actions.

In other words, when most actions belong to common types, aggregate accuracy metrics look impressive — but infrequent actions that often carry critical influence may be systematically missed. The research accordingly argues that action timing and calibration are the true core challenges in policy agent simulation.

This finding carries methodological significance for the field as a whole: chasing aggregate accuracy alone is insufficient for evaluating a simulation system's real capabilities. Evaluation frameworks must be able to expose weaknesses in handling long-tail behaviors.

This problem is known in machine learning as class imbalance, a common trap for classification systems. When a category of events is extremely underrepresented in a dataset, models tend to ignore it in exchange for higher overall accuracy — even when such events are critically important in practice. In the context of policy simulation, "rare stakeholder actions" often correspond to unconventional market interventions, sudden regulatory statements, or institutional moves under extreme sentiment. While exceptional, these actions may be the very triggers for systemic risk. Calibration here refers to whether a model's output probability confidence matches the true frequency of occurrence — a well-calibrated system must not only know "what happened" but accurately estimate "with what probability it happens." Accurate action timing relates to the propagation delay of policy signals and market reaction windows, which is especially sensitive in high-frequency events.

Significance and Outlook

PAWS's value lies not only in its scale but in the auditable substrate it provides. Researchers can use it to evaluate, on the basis of real historical data, how agents influence policy transmission chains, how policy responses cascade, and how behaviors align with ultimate market outcomes.

As LLM-driven agent research rapidly gains momentum, the question of how to anchor multi-agent simulation to the real world — rather than letting it become self-referential — remains unresolved. By binding financial policy events, news evidence chains, and market data together, PAWS offers a pragmatic path toward building credible, reproducible policy simulation research.