Tulip and NVIDIA Build the 'Replayable Factory': Reshaping the Manufacturing Floor with Vision Language Models

The Challenge of 'Playback' in the Factory

Every factory is unique — even when producing the same product, the equipment, workers, and processes all differ. Tulip co-founder and CIO Ronny Kubat noted during a live session that this is precisely why traditional one-size-fits-all manufacturing software struggles to gain traction on the shop floor. Compounding the problem is the fact that factories are in a constant state of flux: products change, workforces change, and lean improvement initiatives demand continuous process refinement.

Traditional manufacturing software systems are highly siloed, and integrating them into a unified whole is both expensive and difficult. When an unplanned stoppage hits due to a stockout, managers have no clear view of what was actually happening on the floor "at that moment." Factory Playback, Tulip's new product, is designed to fill exactly that gap in shop-floor context.

This joint Tulip–NVIDIA technical session, co-presented by Kubat and NVIDIA developer relations AI engineer Alvin Clark, dug into the architectural design of Factory Playback rather than staying at the demo surface.

Giving Manufacturing Systems "Eyes"

The core idea behind Factory Playback is to connect cameras and apply vision language models (VLMs) to inject rich, real-world context into the various digital touchpoints throughout the factory. As Kubat put it, this is like "giving manufacturing systems eyes."

Cameras continuously capture shop-floor activity, processed in real time by VLMs. The outputs can be used for live alerting, triggering agent actions, or retrospective analysis to identify the root cause of on-floor issues.

Take a quality inspection scenario: an operator flags a quality issue within a workflow, which triggers an agent monitoring quality trends. That agent begins searching through data and video to find potential causes, ultimately surfacing the issue to a manufacturing engineer for action. Kubat describes this as "tightening the feedback loop" — dramatically compressing the time between something happening on the floor and being able to act on it.

In the demo, Tulip showcased Playback deployed in a simulated manufacturing environment at its headquarters. Multiple cameras were positioned in 3D space, offering both wide-angle overview shots and close-up views of operator interactions. A timeline viewer at the bottom of the interface overlays video thumbnails, Tulip table records (work order start/stop times, status), and VLM outputs on a single timeline — creating a comprehensive view that functions like a "flight data recorder" for the factory floor.

An Iterable Vision Pipeline: Configure Detection Like Writing a Prompt

One key capability in Playback is letting users build their own vision pipelines. Kubat demonstrated a simple pipeline targeting a filling line: when a person is detected entering the filling zone, the system runs a VLM on that segment with a specific prompt — asking whether a "changeover" is currently in progress — and then requests a structured boolean output from the model.

Based on that structured output, the system writes back to a Tulip table and triggers downstream agent actions.

Even more notable is the prompt validation and iteration mechanism. Users can set expected results using sample videos (including synthetically generated ones), run them in batch, and measure the VLM's pass rate. Because model outputs are stochastic, the same prompt may yield slightly different answers across runs. If performance is unsatisfactory, users can invoke an agent to analyze the test clips and the existing prompt, receive improvement suggestions, and choose whether to accept or reject them. This mechanism brings the "write prompt → evaluate → optimize" engineering workflow directly into factory software.

The Underlying Tech Stack: VSS Blueprint + Cosmos Models

Alvin Clark introduced two core NVIDIA technologies underpinning Playback.

NVIDIA Metropolis VSS (Video Search and Summarization blueprint) addresses a core pain point: the vast majority of factory camera footage just sits there collecting dust, only reviewed manually — at great time cost — after something goes wrong. VSS transforms that footage into an agent that can be queried conversationally, asked questions, and configured to trigger alerts, whether on live streams or archived video.

It is powered by four model types working in concert: a vision language model to understand what's on screen, an NLM for reasoning and response, an embedding model to make content searchable, and computer vision models for fast detection and tracking. On performance: deep retrieval can return results within 5 seconds (including VLM self-critique of its own answers), alerts can trigger within 1 second, and event summarization runs at roughly 60× speed — fast enough to keep pace with live video.

Clark emphasized that VSS is fundamentally a reference architecture, not a ready-to-ship product. It is highly modular — supporting drop-in replacement of detection models, VLMs, embeddings, and databases — and can run at the edge, on-premises, in the cloud, or in hybrid deployments. This design is deliberately shaped for a world where "every factory is different": what counts as an "event" at a welding station is nothing like what counts at a packaging line.

NVIDIA Cosmos is an open frontier omni-model for physical AI. Clark drew an analogy: large language models learned language by reading; physical AI must learn to understand the world — objects, motion, causality, and the difference between "normal" and "anomalous." Cosmos can do four things: reason over video, simulate trainable test worlds, generate image and video data, and drive spatial strategies for robotics.

According to Clark, Cosmos is currently the top-performing open model on industrial visual analytics tasks (dense description and anomaly detection reasoning). As a VLM, it does not merely label people and pallets — it describes in natural language "what is happening," and can reason about cause and effect across time. In a factory context, what happened in the 90 seconds before a stoppage is often more valuable than the stoppage itself. Clark calls this "the ChatGPT moment for the real world." Cosmos is available in three sizes from 4B to 64B parameters, fully multimodal, commercially licensable, and customizable; its Super NanoEdge variant has been optimized for NVFP4 and speculative decoding, enabling it to run on a box sitting on the shop floor rather than requiring cloud compute.

NIM (NVIDIA Inference Microservice) is NVIDIA's packaging standard for inference microservices, bundling models, runtimes, and dependencies into standardized Docker containers so developers can deploy optimized AI models without manually configuring complex inference environments. Each NIM container includes TensorRT optimizations targeted at specific GPU architectures and exposes an OpenAI-compatible API, dramatically lowering the bar for integrating NVIDIA-hosted models into enterprise application stacks. In the Factory Playback context, Cosmos, person detection, privacy masking, and video frame embedding are each delivered as independent NIMs — meaning Tulip can start and stop individual functional modules on demand like microservices, and independently upgrade a single component when NVIDIA releases a new model version without disrupting the rest of the vision pipeline.

Hybrid Cloud Architecture and Privacy Design

Playback uses a hybrid cloud architecture: video is processed entirely locally, on-premises. Tulip provides a rack-mounted server with an NVIDIA GPU for video processing and storage; it also connects to the Tulip cloud, pushing high-level events extracted by the VLM up to the cloud for consumption by automations, tables, and applications.

This architecture offers three advantages: no need to pipe large volumes of camera video bandwidth to the cloud; video never leaves the four walls of the facility, satisfying privacy requirements; and a privacy mode can be enabled to anonymize personnel, keeping the focus on the work itself. When dynamic trigger events spike local compute demand, processing can also be "offloaded" to the cloud as a relief valve.

On the stack side, Playback runs on a K3S cluster on the local server, connecting to local camera streams and an MQTT broker and feeding data into a time-series database. NVIDIA delivers Cosmos plus optimized models for fast person detection, privacy masking, and video frame embedding as NIMs — pre-packaged Docker containers.

For footage retention, Tulip defaults to keeping all video in a ring buffer; capacity depends on disk size, camera count, resolution, and frame rate. If an anomalous event needs to be archived long-term, a pipeline can upload that clip to the cloud for permanent storage.

K3S is a lightweight Kubernetes distribution from Rancher (now part of SUSE), designed for resource-constrained edge computing scenarios. Its binary is under 100 MB and can run on low-power ARM or x86 hardware. Compared to standard Kubernetes, K3S removes certain cloud-provider plugins and consolidates control plane components while maintaining full Kubernetes API compatibility. Deploying K3S in a factory environment means Tulip can manage multiple AI inference services, databases, and message brokers on a local server using container orchestration — supporting rolling updates, health checks, and automatic restarts — without the additional hardware overhead of a full Kubernetes cluster. MQTT is a lightweight publish/subscribe messaging protocol designed for low-bandwidth, unreliable networks; it has become the de facto standard in industrial IoT for connecting PLCs, sensors, and higher-level software systems. Playback uses an MQTT broker to align existing machine data with camera footage on a shared timeline.

Deployment Scenarios and ROI

Playback has been deployed early at several design partners, covering use cases including stoppage root-cause analysis, quality issues (with a goal of real-time alerting), material flow optimization (the "spaghetti diagram" path analysis shown in the demo, serving lean and continuous improvement efforts), and safety compliance (such as detecting PPE usage and boundary violations).

Specialty vehicle manufacturer Terex is a flagship example. It produces highly customized vehicles — aerial work platforms with booms and baskets, for instance — where every unit off the line may be different. For production lines where balancing, floor space utilization, and craftsperson allocation are all extremely difficult to optimize, even marginal productivity gains can have a meaningful impact on margins. Kubat revealed that based on estimates from early partnership work, a single factory could see seven-to-eight-figure ROI.

Person and asset path tracking is powered by NVIDIA DeepStream (a GStreamer-based vision pipeline), running a person detection model on every frame to generate "spaghetti diagrams." Tracking the operational efficiency of forklifts, AMRs, and AGVs across multiple cameras is also a direction VSS continues to evolve toward. For factories that already have cameras, PLCs, and other systems in place, Playback can pull directly from existing camera streams (roughly half of design partners already had camera systems in place) and connect to existing machines and sensors via MQTT, correlating existing time-series and log data with what was actually happening on the floor — which is precisely the value customers prize most.

AMR (Autonomous Mobile Robots) and AGV (Automated Guided Vehicles) are the two primary workhorses of modern factory logistics automation. AGVs rely on fixed guidance paths (magnetic tape, QR codes, or laser reflectors) and are costly to reroute; AMRs use SLAM (Simultaneous Localization and Mapping) technology to autonomously plan paths and navigate around obstacles, offering greater deployment flexibility. Both types of equipment typically share aisles with forklifts and operators in large manufacturing facilities, making their actual operational efficiency — wait times, detour frequency, path congestion — difficult to reconstruct from sensor logs alone. Playback's multi-camera tracking of these assets' "spaghetti diagram" trajectories lets lean teams compare visually observed actual paths against theoretical optimal routes, providing improvement evidence grounded in real shop-floor conditions rather than manual time studies.