Using Large Language Models to Break Through the Scalability Bottleneck in Interpretive Structural Modeling

When Traditional Decision Modeling Meets Scalability Challenges

Interpretive Structural Modeling (ISM) is a classic method in the field of multi-criteria decision making. What sets it apart from other methodologies comes down to three core capabilities: modeling causal relationships, making binary-scale judgments between factors, and producing a hierarchical structural representation. This approach is widely used in systems engineering, supply chain management, policy analysis, and other domains to help decision-makers untangle complex dependency relationships among variables.

But ISM has a long-standing pain point. The traditional modeling process relies on repeated interactions with domain experts — researchers must continually ask experts questions like "Does factor A influence factor B?" until consensus is reached on all judgments. This process is not only lengthy and labor-intensive, but more critically, it severely limits ISM's scalability. When a study involves dozens or even hundreds of variables, the number of factor pairs requiring expert judgment grows quadratically, making it nearly impossible to complete within a reasonable timeframe.

The core operation in ISM is constructing a Reachability Matrix: researchers first build an n×n binary matrix where each cell indicates whether "factor i can influence factor j," with values of 0 or 1. After filling in this matrix, indirect influence relationships are identified through the transitive closure operation (repeated Boolean matrix multiplication), and all factors are ultimately ranked by influence into a pyramid-like layered digraph. This hierarchical diagram intuitively shows which factors are the driving roots of the system (bottom layers) and which are passive outcome variables (top layers). When the number of variables is n, experts may need to evaluate up to n×(n-1) factor pairs — the mathematical root of the scalability problem. Fifty variables means roughly 2,450 expert judgments; 100 variables requires nearly 9,900.

Treating LLMs as "Imperfect Experts"

This new paper on arXiv proposes a straightforward solution: integrating large language models into the ISM workflow to replace or assist human experts in making causal judgments.

The work builds on recent research into "causal graph discovery using LLMs." The core idea is that LLMs can be treated as "imperfect experts" — they possess extensive domain knowledge and can make reasonable inferences about causal directions between variables. While not perfectly accurate, they can handle the volume of judgments that human experts simply cannot sustain, at scale and low cost.

This framing is crucial. Rather than treating LLMs as perfect causal reasoning machines, the research acknowledges that their outputs contain errors, then evaluates the degree to which they can reconstruct the hierarchical structure required by ISM. This pragmatic stance gives the overall approach stronger practical engineering value.

Comparing Four Graph Discovery Methods

The central contribution of the research is a systematic comparison of four causal graph discovery strategies at different levels of granularity:

Pairwise

The finest-grained approach, where the model evaluates whether a causal relationship exists between one pair of factors at a time. This most closely resembles the traditional ISM expert Q&A format, but still requires a large number of queries at scale.

k-wise

The model evaluates relationships among several factors at once — a middle ground between pairwise and rowwise approaches.

Rowwise

For each factor, the model judges its relationship with all other factors at once, effectively filling in an entire row of the adjacency matrix in a single query.

Full graph

The coarsest-grained approach, where the model outputs the entire causal graph structure in a single inference step.

These four methods represent different tradeoffs between query efficiency and accuracy: the coarser the granularity, the fewer model calls required and the higher the efficiency — but the greater the difficulty of each individual inference, which may impact accuracy.

Evaluation Results: Rowwise and Full Graph Methods Win

The study used two key metrics to evaluate performance: SHD (Structural Hamming Distance, which measures structural differences between the generated graph and the ground truth — lower is better) and F1-score (combining precision and recall — higher is better).

Results showed that the Rowwise method was among the best performers, achieving SHD=160 and F1-score=0.77; the Full graph method performed better on structural distance, reaching SHD=135 and F1-score=0.73.

This finding is illuminating. It suggests that allowing LLMs to reason holistically about causal structure within a larger context (rowwise or full graph) tends to outperform isolated pairwise judgments. This is likely because a global perspective helps the model grasp the systemic relationships between variables, whereas pairwise judgments are prone to losing context and producing contradictory conclusions. For practical applications, these two methods also offer better accuracy and call efficiency — precisely what ISM needs to scale to large numbers of variables.

Structural Hamming Distance (SHD) is a standard metric for comparing graph structures. It counts how many edges need to be added or removed to make two graphs identical — including falsely added edges (false positives), missing edges (false negatives), and edges with reversed direction. SHD=0 means perfect reconstruction; higher values indicate greater deviation. In this study, SHD=135 (full graph method) represents an acceptable but improvable error level relative to the total possible edges in the full adjacency matrix. This becomes more intuitive when combined with the F1-score: F1=0.77 means the model achieves 77% combined precision and recall on the binary classification task of determining "whether a causal edge exists between two factors." This is already a strong result for an unsupervised, zero-shot causal discovery task — but still falls short of what would be needed for deployment in high-stakes decision making.

What This Research Means

From a methodological standpoint, the LLM-ISM integration breathes new life into a classic decision modeling tool. It directly addresses ISM's long-standing structural limitation of being "difficult to scale," making it potentially applicable to complex systems research involving hundreds of variables — no longer constrained by the human bottleneck of expert interviews.

Of course, the research honestly confronts its limitations. Framing LLMs as "imperfect experts" itself acknowledges that current causal judgments still contain errors, and F1-scores in the 0.73–0.77 range indicate considerable room for improvement. In high-stakes decision scenarios, LLM outputs are better suited as an initial screening or assistive tool rather than a complete replacement for final human expert judgment.

For practitioners in systems engineering, operations research, and causal inference, this work points to a direction worth watching: how to get large language models to work in concert with mature structured methodologies — preserving the rigor of the latter while leveraging the scalability of the former. As LLMs continue to improve their causal reasoning capabilities, the accuracy of such hybrid approaches is likely to improve further.