Can LLMs Read Fraud Networks? A Gemma 4 31B Test

Can LLMs Read Fraud Networks? A Gemma 4 31B Test

Using graph neural networks for fraud detection is already a relatively mature approach, but a more fundamental question has long lacked a direct experimental answer: if you translate the relational network between transactions into text and feed it to a large language model, can it determine from that text description alone whether a transaction is suspicious?

A developer built such a benchmark task in the Kaggle Benchmarking Challenge, using the IEEE-CIS fraud detection dataset to test Gemma 4 31B (google/gemma-4-31b). The conclusion is somewhat sobering: the model's performance is only slightly better than a coin flip, and clearly lags behind the GraphSAGE graph neural network baseline on the same task.

Benchmark Design: Turning Graph Structure into Text

The core idea of this benchmark is not to have the model read graph data directly, but to convert the target transaction and its local network neighborhood into a text description.

Specifically, each sample contains a target transaction and several neighboring transactions connected to it through shared attributes (card number, address, device information, etc.). The prompt presents textual relational information about these transactions, without including ground-truth labels. The model needs to complete two tasks:

• Classify the target transaction as fraudulent or legitimate
• Identify a potentially suspicious transaction from the neighboring transactions

This design attempts to answer a clear question: can a general-purpose language model extract useful structural information from a "graph context → text" conversion?

GraphSAGE is a class of inductive graph neural networks proposed by Hamilton et al. in 2017, short for Graph SAmple and aggreGatE. Its core idea is: for a target node, sample a fixed number of neighbors (rather than using all neighbors), then use aggregation functions (such as mean, LSTM, pooling) to progressively gather neighbor features into the node itself, thereby generating a node representation at each layer that fuses local neighborhood information. Unlike graph convolutional networks that require full-graph training, GraphSAGE's sampling mechanism allows inference on unseen nodes, making it very suitable for scenarios like transaction graphs where nodes are continuously added and the structure evolves dynamically. On tabular data like IEEE-CIS, the typical approach is to treat each transaction as a node, shared attributes (card number, address, device) as edges, and then attach a classification head to the GraphSAGE-learned embeddings for fraud discrimination. Precisely because of this, GraphSAGE naturally "sees" the real graph topology, whereas the LLM only sees the topology after it has been serialized into text—the two do not start from an equal information footing.

Test Results: What Does 55% Classification Accuracy Mean?

On 100 samples, Gemma 4 31B achieved a classification accuracy of 55% and a neighborhood suspicious transaction identification (proxy metric) accuracy of 53%.

Among the 100 labels in the classification task, fraudulent and legitimate each accounted for 50, so the majority-class baseline is exactly 50%. A score of 55/100 is statistically insufficient to prove that the model has discriminative ability beyond the baseline. The developer also explicitly stated that the results do not support the conclusion that the model performs better than the reference baseline.

For comparison, consider the GraphSAGE graph neural network baseline: on the same 100 samples, GraphSAGE got 67 correct, while Gemma got 55. Their cross-performance breakdown is:

• Both correct: 44
• Gemma correct but GraphSAGE wrong: 11
• GraphSAGE correct but Gemma wrong: 23
• Both wrong: 22

On this batch of samples, the graph neural network baseline's advantage is clear.

When evaluating binary classification models, the majority-class baseline is the minimum reference point: when positive and negative samples are split evenly, a model that always predicts "legitimate" can still achieve 50% accuracy. Therefore, on a balanced dataset, whether the 5-percentage-point gap between 55% and 50% is significant depends on sample size and statistical testing. With n=100, a single sample represents 1 percentage point, and the 95% confidence interval is roughly 45%–65%, fully covering 50%—meaning these 5 percentage points are very likely random fluctuation rather than a real signal. To determine whether a model truly outperforms the baseline, one typically needs a larger sample (hundreds to thousands), a paired test (such as the McNemar test, specifically designed to compare the differences between two classifiers on the same batch of samples), and repeated experiments under multiple random seeds. What is honest about this benchmark is precisely this: it does not package 55% as "beating the baseline," but explicitly acknowledges the insufficiency of statistical evidence.

Runtime Details and a Notable Engineering Issue

This run produced 200 non-empty responses in total (100 each for classification and neighbor identification), and the task executed completely.

However, an engineering detail emerged during the process: in the control probe, when a 128-token limit was sent nested under the extra_body parameter, the model returned empty responses; after passing max_tokens as a top-level parameter, all 200 responses were produced normally. This phenomenon is worth noting for developers using similar API interfaces.

The Boundary of the Neighbor Metric: A Proxy Score Is Not Real Capability

The 53% on the neighbor identification task needs to be interpreted with particular caution. The IEEE-CIS dataset does not provide verified fraud ring membership labels or ring leader labels, so the developer settled for using the fraud label of the selected neighboring transaction itself as the scoring basis.

This means 53% is a fraudulent neighbor proxy score, and cannot be treated as evidence that the model can identify real fraud rings or their leaders. This distinction is crucial when interpreting the results—treating a proxy metric as proof of capability is a common misreading trap in fraud detection research.

A proxy metric refers to using another available and relevant quantity to approximate the target concept when ground-truth labels for that concept are lacking. Here, the target concept is "fraud ring member or leader," but the IEEE-CIS dataset only has per-transaction isFraud labels and does not provide verified ring membership annotations. Therefore, the developer could only score the neighbor identification task using "whether the selected neighboring transaction itself is fraudulent." The problem is that fraudulent transactions do not necessarily have real criminal connections—two independent card fraud incidents may be connected because they share a high-risk address or device fragment. Using the neighbor's fraud label as a proxy actually measures "whether the model can pick out a neighbor that happens to be fraudulent," rather than "whether the model can identify nodes that are truly colluding with the target transaction." In fraud detection research, treating such proxy scores as evidence of ring identification capability is a common misreading; any conclusion based on a proxy metric should clearly state what it measures and what it does not.

Conclusion and Next Steps

The main finding of this benchmark can be summarized as: providing the local transaction neighborhood to Gemma 4 31B in text form did not produce strong predictive performance at this sample size.

The developer's proposed follow-up plans include: testing more models, expanding the sample size, and running prompt ablation experiments to quantify the actual impact of neighborhood details on prediction results.

For researchers and engineers concerned with the boundaries of applying LLMs to graph-structured tasks, the value of this experiment is not in proving that "LLMs can't do it," but in providing a reproducible testing framework and an honest negative result—in a field dominated by GNNs, a general-purpose language model relying solely on textualized relational descriptions currently struggles to match specialized graph models. It should be noted that this is a single-model evaluation, not a large-scale cross-model ranking, and the scope of the conclusions should be limited to the model and samples tested here.

One-line Summary

An experiment shows that Gemma 4 31B, using only textualized transaction network descriptions for fraud classification, achieves 55% accuracy—nearly random—clearly trailing GraphSAGE's 67%.

Paragraph Summary

The benchmark converts the target transaction in the IEEE-CIS fraud dataset and its neighboring transactions connected by shared attributes (card number, address, device) into text, then asks Gemma 4 31B to judge whether the transaction is suspicious and identify potentially suspicious neighbors, with no ground-truth labels in the prompt. On 100 balanced samples, the model's classification accuracy is 55%, only slightly above the 50% majority-class baseline, which is statistically insufficient to prove it beats the baseline; the GraphSAGE graph neural network baseline on the same task got 67 correct, uniquely claiming 23 correct samples beyond the 22 that both got wrong. The 53% on the neighbor identification task is only a fraudulent neighbor proxy score and, because the dataset lacks real ring labels, cannot be regarded as ring identification capability. There is also an engineering finding: when max_tokens is a top-level parameter, all 200 responses are normal, but when nested under extra_body, empty responses are returned.