Millimeter-Wave Radar Point Cloud Object Classification: MLP in Practice and Key Insights

Introduction: The Object Classification Challenge with Radar Point Clouds

In autonomous driving perception systems, cameras and lidar often steal the spotlight, but millimeter-wave radar remains an indispensable component of the perception stack thanks to its all-weather capability and resilience to harsh conditions. Millimeter-wave radar typically operates at the 77GHz band (wavelength ~4mm), directly measuring a target's range, velocity, and angle. Compared to cameras, radar is independent of lighting conditions and performs reliably at night, under glare, or against backlight. Compared to lidar, radar suffers minimal attenuation in rain, snow, fog, and dust, with detection ranges exceeding 200 meters. However, radar's angular resolution is far lower than lidar's (a traditional 3Tx/4Rx antenna array has an angular resolution of roughly 15°), resulting in extremely sparse point clouds — a pedestrian might correspond to just 1–3 detection points, whereas lidar at the same range would produce dozens to hundreds. This "sees far but sees blurry" characteristic means radar typically plays the role of "initial screening + velocity anchoring" in perception fusion architectures, with classification historically being its weakest link. How to accurately identify object classes from such limited point clouds is a long-standing engineering challenge.

Recently, a radar signal processing engineer shared on Reddit a five-class object recognition model he trained on the RadarScenes dataset. While structurally simple — using only a three-layer MLP (multilayer perceptron) — the depth of his analysis on data fundamentals and model limitations provides extremely valuable reference for radar machine learning practitioners.

Method Overview: Histogram Features and a Lightweight MLP Architecture

The project's goal is to classify objects from a single radar scan into five categories: car, large_vehicle, two_wheeler, pedestrian, and pedestrian_group.

Input Feature Design

The author adopted a histogram-based feature encoding approach, converting each scan's point cloud data into a 16-bin histogram vector. This idea originates from the paper Histogram-based Deep Learning for Automotive Radar. The core concept of histogram feature encoding is to divide the value range of a physical quantity (such as RCS, radial velocity, azimuth angle, etc.) across all detection points associated with an object into a fixed number of bins, count the number of points falling into each bin, and form a fixed-length vector representation. For example, if the RCS value range is divided into 16 bins, then regardless of whether an object has 2 or 20 detection points, it is ultimately mapped to a 16-dimensional vector. This method is essentially a discrete approximation of point set distributions, preserving more distribution shape information — such as bimodal distributions, skewed distributions, etc. — compared to per-point statistics (mean, standard deviation, etc.). It is particularly practical in radar scenarios because the number of detection points per radar target varies enormously, and histograms naturally solve the classic point cloud processing problem of "inconsistent input dimensions." Compared to processing raw point clouds directly, histogram features offer fixed dimensionality and insensitivity to point count, making them ideal as input for traditional fully-connected networks.

The network itself is a 3-layer MLP, with class-weighted cross-entropy as the loss function to mitigate bias caused by data imbalance. In standard cross-entropy loss, each sample contributes equally to the total loss, meaning majority classes (such as cars) dominate the gradient update direction while minority classes (such as two-wheelers) are effectively "drowned out." Class-weighted cross-entropy corrects this bias by assigning each class a weight inversely proportional to its frequency — for example, if car samples outnumber two-wheeler samples by 10x, the per-sample loss weight for two-wheelers is set to 10x that of cars. This approach is widely used in class-imbalanced classification tasks, though excessively high weights can lead to overfitting on minority classes. Other strategies for addressing imbalance include oversampling (e.g., SMOTE), undersampling, Focal Loss, and others, each with its own applicable scenarios. The author deliberately scoped the project to single-frame scans, with multi-frame accumulation listed as a future research direction.

Inherent Challenges of the RadarScenes Dataset

Before presenting the experiments, the author candidly listed several structural issues with the RadarScenes dataset. RadarScenes is a large-scale automotive radar dataset released by Mercedes-Benz, containing approximately 4 hours of driving scenario data covering urban roads, rural roads, and highways. Data was collected using four 77GHz radar sensors providing 360-degree coverage around the vehicle. The dataset provides point-level semantic annotations with 11 original classes, though due to extremely small sample sizes for some classes, researchers typically merge them into 5–6 classes in practice. The dataset's unique value lies in providing complete raw radar measurements — including range, radial velocity, RCS, and azimuth angle — rather than post-processed object-level outputs, allowing researchers to directly explore point-cloud-level classification algorithms.

Specific challenges include:

• Class imbalance: Two-wheelers and large vehicles have significantly fewer samples;
• Class merging: Due to data scarcity, two_wheeler combines bicycles and motorcycles, while large_vehicle merges trucks, buses, and even trains;
• Sequence bias: Long trajectories of slow-moving objects can skew the velocity distribution of a particular data split, causing dramatic F1 score fluctuations between different cross-validation folds.

Core Finding: Detection Point Count Determines the Classification Performance Ceiling

The most valuable insight from this work is the revelation of a strong correlation between the number of detection points per object and classification performance.

The author bucketed validation set predictions by the number of detection points per object and computed Macro F1 for each bucket. The results show that as the number of radar detection points per object increases from 1 to 5, Macro F1 jumps from 0.381 to 0.764 — nearly doubling.

This finding strikes at the core contradiction of radar perception: sparsity. When an object reflects back only a handful of points, no matter how complex the model, the available information itself is insufficient to support reliable classification. This also explains why the author's series of ablation experiments (larger MLPs, different feature encodings, different histogram binning strategies) produced smaller performance changes than simply altering the train/validation/test split. In other words, when the radar data itself is the bottleneck, the room for model architecture tuning is very limited.

You might have missed that the author also tried replacing histograms with per-instance statistics (mean, median, standard deviation), which actually slightly degraded performance — confirming that the distributional representation of histograms does capture richer information.

Confusion Analysis: Radar Physics Defines Classification Boundaries

The model's varying performance across classes is essentially a direct reflection of radar physical characteristics.

Best and Worst Performing Classes

Car and pedestrian achieved the best classification results, while two_wheeler performed the worst. There are clear physical explanations behind this:

• Cars misclassified as large vehicles: When a car has an unusual width, or when multipath effects cause its RCS (Radar Cross Section) to be abnormally high, the model tends to classify it as a large vehicle. RCS is a physical quantity measuring a target's ability to reflect radar energy, expressed in square meters (dBsm), intuitively understood as the "equivalent reflective area of a target as seen by radar." Typical RCS values differ significantly across object classes: a car's RCS is usually 10–100 m² (10–20 dBsm), while a pedestrian's is about 0.5–2 m² (-3 to 3 dBsm), and a bicycle's is even smaller. However, RCS is not a fixed value — it varies dramatically with target orientation, material, surface shape, and observation angle. For example, a truck facing the radar sideways might have an RCS of 200 m², but only 20 m² when facing head-on. Multipath effects (signals reflecting off the ground before reaching the target) can inflate RCS measurements or create spurious detection points, which is one physical reason cars get misclassified as large vehicles.

• Two-wheelers misclassified as pedestrians: The vr_compensated (compensated radial velocity) distributions of the two classes overlap heavily, and this is precisely the most important feature for distinguishing them. Radar directly measures the radial velocity component of a target relative to the radar via the Doppler effect — that is, the velocity projection along the radar's line of sight. However, since the radar is mounted on a moving vehicle, the raw measurement mixes in the ego-vehicle's motion contribution. vr_compensated is the target's true radial velocity after ego-motion compensation — obtained by decomposing vectors using the ego-vehicle speed and target azimuth to remove the ego-motion component. This feature is critical for classification: cars typically move at 20–120 km/h, pedestrians at 0–6 km/h, and bicycles at 5–25 km/h. But when a target is stationary or moving perpendicular to the radar's line of sight, vr_compensated approaches zero, at which point the velocity features of different classes completely overlap and the model loses its strongest discriminating cue. A stationary or idling two-wheeler is virtually indistinguishable from a pedestrian in the radar's eyes.

A Typical Misclassification Case

The author provides a highly representative example: in one scene, a nearly stationary two-wheeler with only a single detection point was predicted as a pedestrian — because its velocity was close to zero, indistinguishable from a pedestrian. Yet in the same scene, a car with equally just a single point was correctly classified, because its RCS and Doppler characteristics were sufficient for differentiation.

This comparison vividly illustrates: when velocity information fails, the model can only rely on RCS and other physical quantities; and for targets like two-wheelers with small RCS and variable speeds, once velocity becomes unavailable, classification becomes extremely difficult.

Future Improvements and Practical Takeaways

Addressing the fundamental bottleneck of radar sparsity, the author outlined two clear improvement paths:

1. Introduce spatial encoding schemes, such as PointNet-style architectures that directly process point clouds, to better leverage spatial geometric information. PointNet is a pioneering deep learning architecture proposed by Stanford University in 2017, the first to enable direct processing of unordered point sets without converting point clouds into voxels, meshes, or images. Its core design includes two key mechanisms: first, per-point MLPs extract high-dimensional features for each point; second, a symmetric function (max pooling) aggregates global features, ensuring the network output is invariant to the ordering of input points (permutation invariance). The potential advantage of introducing PointNet-style architectures in radar scenarios is that they preserve each detection point's spatial coordinate information (range, angle), whereas histogram features lose inter-point spatial relationships during the aggregation process. The subsequent PointNet++ further introduced hierarchical local feature learning, making it more suitable for processing point clouds with non-uniform density — which aligns well with radar point clouds' characteristic of being denser nearby and sparser at distance.

2. Accumulate multiple scan frames, alleviating single-frame sparsity through temporal information stacking, and further explore micro-Doppler features — which are particularly valuable for distinguishing pedestrian limb motion from two-wheeler wheel rotation characteristics. Micro-Doppler refers to the additional modulation features in the Doppler spectrum caused by micro-motions (vibration, rotation, swinging, etc.) of a target's constituent parts beyond its overall translational movement. For pedestrians, limb swinging produces characteristic "butterfly-shaped" broadening around the main Doppler frequency (known as the micro-Doppler signature); for bicycles, wheel rotation produces periodic micro-Doppler modulation. These features form unique "fingerprints" in the time-frequency domain — even when targets have similar overall velocities, their micro-Doppler signatures may be distinctly different. Extracting micro-Doppler features typically requires performing Short-Time Fourier Transform (STFT) on raw radar ADC data or range-Doppler maps, which demands access to lower-level radar data than detection-point level and requires multi-frame accumulation to achieve sufficient temporal resolution — which is why the author listed it as a research direction following "multi-frame accumulation."

Key Takeaways for Radar Machine Learning Practitioners

The significance of this work lies not in how advanced the model is, but in the rigor of its methodology. It reminds us that:

• In sparse data scenarios such as radar, data split sensitivity may far exceed the gains from model tuning; multi-fold cross-validation is essential during evaluation;
• Performance bottlenecks often stem from the physical nature of the data (e.g., sparsity, feature overlap), not model capacity;
• Deeply understanding the physical reasons behind misclassifications provides far better guidance for next-step improvements than blindly stacking network layers.

For any engineer working in radar machine learning, this complete experimental record serves as a pragmatic and honest reference.

Key Takeaways