Real-Time AI Decision Model Benchmark: How Latency Determines the Winner

In real-time AI applications, a model that makes more correct decisions isn't necessarily the better choice — it might simply be slower. The Hopsworks team used a computer game inspired by Subway Surfers to benchmark three pre-trained "decision" models (essentially classifiers), and the results exposed a core issue that's often overlooked in real-time AI systems: latency.

Test Setup: A Game That's Faster Than the Models

The benchmark's core premise is straightforward — to run farther in the game, you must make correct decisions faster. The runner faces randomly generated obstacles that begin appearing at preset distance thresholds. Each model receives the same underlying game state, formatted according to its own interface spec:

{ lane, airborne, ahead }

The ahead field describes the rows ahead, their distances, and obstacles in each lane. The model must decide how to act before the runner arrives.

The runner starts at 45 m/s, accelerating at 1.6 m/s per second up to a maximum of 160 m/s. Beyond roughly 5,000 meters, the game becomes brutally unforgiving and survival increasingly depends on luck. In other words, the faster the runner moves, the narrower the reaction window for the model — making this a ruthless stress test for real-time decision-making.

Head-to-Head Results

Three pre-trained decision models were evaluated: Kumo, Jev, and Qwen. Here are the results:

Model	Runs	Avg Distance	Median Distance	Best Run
Kumo	500	2,119 m	—	—
Qwen	500	1,318 m	—	—
Jev	30	—	540 m	1,417 m

Kumo clearly led with an average distance of 2,119 meters. Qwen came in second at 1,318 meters. Jev, over 30 runs, achieved only a 540-meter median with a best run of 1,417 meters.

The deployment setup is worth highlighting: Kumo and Qwen are local models hosted on Hopsworks, with the game running in the same environment. Jev, by contrast, is a remote model hosted in a separate data center. This deployment difference turned out to be the decisive variable.

Why Kumo Won

Kumo's victory isn't mysterious: it achieved lower decision latency and lower network latency while maintaining solid decision quality. Compared to Qwen, which also runs on CPU, Kumo consistently delivered higher average distances.

A natural follow-up question is: would switching to a GPU help? The team's answer was "possibly," but for small, latency-sensitive requests, the additional infrastructure overhead might cancel out the raw compute gains. This remains an untested hypothesis. It actually highlights a counterintuitive aspect of real-time inference — more powerful hardware doesn't necessarily mean faster end-to-end response, because scheduling, data transfer, and cold-start overhead all count toward total latency.

CPU inference can sometimes outperform GPU for small models because GPU latency has more moving parts: data must be copied from host memory to device memory (Host-to-Device transfer), CUDA kernel launches carry their own fixed overhead (kernel launch latency), and batching schedulers introduce queuing delays. For small classifiers with limited per-inference computation, these fixed costs represent a large fraction of total latency — meaning the GPU's throughput advantage doesn't translate into lower single-request latency. CPU inference, by contrast, has a shorter, more direct execution path and tends to be more consistent in scenarios that require processing one request at a time with extreme latency sensitivity. This stands in sharp contrast to batch inference, where GPU parallelism can be fully leveraged.

Why Jev Fell Behind: Latency Consumed the Decision

Jev's bottleneck is clear — round-trip latency exceeding 200 milliseconds. Around the 1,500-meter mark, the game speed becomes high enough that these responses become meaningless.

A simple calculation explains why: at the maximum speed of 160 m/s, 200 milliseconds means the runner has already traveled approximately 32 meters while waiting for a decision. No matter how correct the decision, by the time it arrives, the game state has already changed. This is the fatal flaw of remotely hosted models in real-time scenarios — the physical distance and network round-trip create a hard latency floor that cannot be bypassed.

Network Round-Trip Time (RTT) is the sum of multiple stacked delays: a request leaves the client, travels through the local network and internet backbone, enters the target data center, reaches the inference server, completes computation, and returns along the same path. Even if the model inference itself takes only a few milliseconds, the physical speed-of-light propagation across data centers (roughly 5ms per 1,000 km one-way), TCP handshakes and congestion control, and cloud provider load-balancing layers can push total RTT to tens or even hundreds of milliseconds. This is why co-location is the golden rule for real-time inference — deploying the model in the same data center, or even the same process, as the caller can compress network latency from "cross-city scale" down to "microsecond scale," fundamentally changing the latency profile.

Lessons for Real-Time AI: Use Reaction Deadlines as Your Benchmark

The biggest takeaway from this test is that evaluating real-time AI models requires looking at the entire decision loop, not just decision quality alone. The team identified three key dimensions:

• Decision quality: Does it choose the correct action?
• End-to-end latency: Does that action arrive in time?
• Deployment fit: Can your infrastructure support both of the above simultaneously?

A model that leads only on accuracy but can't return results within the deadline is worthless in a real-time system. Latency is not a secondary metric to be optimized later — it should be the primary constraint in your selection criteria.

The entire benchmark — including all models and the game itself — ran on Hopsworks' own office infrastructure, fully self-hosted, with no game data sent to the cloud. This setup itself reflects the common requirements of real-time and data-privacy-sensitive deployments.

In this particular benchmark, Kumo was the top performer. But the broader conclusion is this: always evaluate models against your own application's reaction deadline, rather than trusting scores on generic leaderboards.

Note: This article is based on test data from a single source (a Reddit post). The test scale is limited (Jev had only 30 runs) and deployment environments differed between models. Conclusions should be interpreted with appropriate caution.