Restaurant CV System in Practice: Architecture Design for Occupancy and Table Cleanliness Detection

In the restaurant industry, table turnover rate and customer experience directly impact revenue. A developer is seeking architectural advice on Reddit for building a production-grade computer vision system to detect seat occupancy and table cleanliness in restaurants. This is not a simple demo project but an engineering challenge aimed at real-world deployment. Based on that discussion, this article breaks down the technical difficulties, architectural choices, and functional design ideas to provide a reference for CV implementation in similar scenarios.

Core Task Breakdown: Two Vision Problems, Two Technical Paths

The system needs to solve two problems simultaneously: seat occupancy detection and table cleanliness assessment. The former is essentially object detection and state classification—identifying whether there are people in table and chair areas; the latter is closer to fine-grained image classification or semantic segmentation—determining whether there are dishes, trash, or stains on the table.

According to the original post, these two tasks share the same camera input but have different output logic: occupancy detection requires real-time performance (for dynamic turnover statistics), while cleanliness detection can tolerate some latency (for cleaning scheduling). This means the architecture needs to support multi-task parallel inference and be able to configure different processing frequencies per task.

Architecture Recommendation: Hybrid Edge Preprocessing + Cloud Inference

The core constraints in production are cost and reliability. A pure cloud solution puts heavy pressure on bandwidth, while a pure edge solution is limited by compute power. Multiple practitioners in the comments recommended a hybrid architecture:

• Edge side: Deploy lightweight devices locally in the restaurant (such as Jetson Nano or NPU-equipped boxes) responsible for video frame sampling, ROI cropping (keeping only table and chair areas), and preliminary filtering (such as discarding frames with no people), greatly reducing the amount of uploaded data.
• Cloud side: Receive cropped images or feature vectors uploaded from the edge, run heavier detection models, and aggregate multi-camera data for global state management.

Some developers pointed out that ROI cropping is the key optimization—in restaurant scenes, table and chair areas usually occupy only a small portion of the frame, and cropping can reduce transmission volume by an order of magnitude. At the same time, the edge side should cache the most recent frames so that basic detection can be maintained during network interruptions.

Model Selection and Data Strategy

For occupancy detection, the mainstream choices are real-time detectors such as the YOLO series or RT-DETR, combined with a simple classification head to determine "person/no person." Cleanliness detection is more challenging: table state is a continuously changing spectrum, with no clear boundary between "clean" and "messy."

The original poster mentioned planning to use a semantic segmentation model (such as SegFormer) to identify objects on the table (plates, cups, napkins) and then determine cleanliness status through rules or a small classifier. However, some comments argued that directly training a binary classifier (clean/not clean) may be more practical, provided there are enough labeled samples.

At the data level, annotation costs in restaurant scenes are high, and there are privacy concerns. A semi-automated annotation workflow is recommended: first use pretrained models to generate candidate boxes, then manually correct them; at the same time, blur customer faces and retain only body contours and table areas.

YOLO (You Only Look Once) is a representative single-stage object detector. Its core idea is to transform detection into a regression problem, directly outputting bounding boxes and class probabilities through a single forward pass, resulting in extremely fast inference suitable for real-time scenarios. RT-DETR (Real-Time Detection Transformer) is based on the Transformer architecture and uses attention mechanisms to model global context, improving detection accuracy while maintaining real-time performance, especially for occlusion and small-object scenarios. Both can be used for occupancy detection; YOLO has a more mature deployment ecosystem, while RT-DETR performs more robustly under complex lighting. SegFormer is a Transformer-based semantic segmentation model with a hierarchical encoder structure that can handle arbitrary-resolution inputs without positional encoding and excels at capturing fine-grained regions. In this system, SegFormer can be used to identify dishes, trash, and other items on tables pixel by pixel, providing richer spatial information for cleanliness assessment, but its computational overhead is higher than that of classification models.

Functional Design: From Detection to Actionable Operational Insights

Simply outputting "person/no person" has limited value. Several high-value functional directions emerged in the discussion:

1. Turnover time prediction: Combine occupancy detection and table history data to predict when a table is about to become free and remind staff to prepare in advance.
2. Cleaning priority ranking: Generate a cleaning task queue based on table messiness and vacancy duration, avoiding blind patrols by staff.
3. Area heatmaps: Count occupancy density in different areas during different time periods to assist with staffing and table layout adjustments.
4. Anomaly alerts: Trigger immediate reminders for events such as customers occupying a table for an extended period (which may affect turnover) or suspected spilled liquids on the table.

These functions require the system not only to output detection results but also to maintain a table state machine (idle/occupied/pending cleaning/cleaning in progress) and integrate with the restaurant's POS or scheduling system.

Hidden Challenges in Production Environments

The original post's title emphasizes "production," meaning it must face real-world problems beyond the lab:

• Lighting variation: Restaurant lighting is complex; backlight, shadows, and night modes can all affect detection stability. It is recommended to simulate various lighting conditions in data augmentation or use models robust to lighting.
• Occlusion: Customers' bodies occlude tables, and chairs occlude table legs, resulting in incomplete detection boxes. Temporal information can be introduced to fill in gaps using previous and subsequent frames.
• Camera viewpoint differences: Installation angles and heights differ across restaurants, so the model needs some viewpoint generalization capability or support for rapid fine-tuning.
• Privacy compliance: Since customer imagery is involved, anonymization must be completed on the edge side to avoid uploading raw video to the cloud.

The table state machine is the core abstraction of operational logic. It models each table as a finite set of states (idle, occupied, pending cleaning, cleaning in progress) and defines state transition conditions. For example, when occupancy detection confirms people are present across multiple consecutive frames and the cleanliness model determines the table is clean, the state transitions from idle to occupied; when people leave and the table is messy, it transitions to pending cleaning; after the cleaning task is completed, it returns to idle. The state machine needs to handle state fusion across multiple camera viewpoints to avoid state jitter caused by occlusion or false detections. Typically, time windows and confidence thresholds are introduced for smoothing, and the latest state snapshot of each table is maintained. After integration with the POS system, the state machine can associate order information to enable turnover time prediction and automatic dispatch of cleaning tasks, serving as a key intermediate layer connecting visual perception and business actions.

Summary and Open Questions

This discussion presents a typical CV implementation scenario: clear tasks but fuzzy boundaries, technically feasible but engineering-complex. Architecturally, hybrid edge-cloud mode, ROI cropping, and multi-task scheduling are consensus directions; model-wise, occupancy detection is relatively mature, while cleanliness detection still requires exploration of more effective representation methods.

The original poster is still seeking more architectural advice, especially on how to handle multi-camera synchronization and state consistency. For teams planning to enter similar fields, it is recommended to start with a pilot in a single store with a small number of cameras, validate detection accuracy and operational value, and then gradually expand.