Optimizing SegFormer-B2 on Jetson AGX: Boosting FPS and Reliability for Off-Road Semantic Segmentation

In autonomous driving and robotics, semantic segmentation in off-road environments has always been a tough nut to crack. Unlike structured urban roads, outdoor scenes lack clear lane markings, curbs, and standardized object categories. Models must accurately distinguish traversable regions from obstacles while maintaining sufficient real-time performance on resource-constrained edge devices. A recent engineering write-up shared by a developer on Reddit puts the typical challenges of this problem squarely on the table.

Project Background: Off-Road Navigation on Edge Hardware

The goal of this project is to build an autonomous off-road navigation system. The core tech stack is fairly representative:

• Model: SegFormer-B2, trained on the Yamaha-CMU Off-Road (YCOR) dataset, achieving ~57.4% mIoU on the validation set
• Hardware: Jetson AGX + ZED 2i stereo camera
• Inference: TensorRT FP16 quantization, 512×512 input resolution, current frame rate of 20–30 FPS
• Goal: First reach 40–50 FPS, ultimately aiming for 60+ FPS

The developer also tested lighter architectures like Fast-SCNN and PIDNet-S, which are faster but noticeably less accurate than SegFormer-B2. This reveals the eternal trade-off in edge-side semantic segmentation — speed versus accuracy. The SegFormer family, as a Transformer-based segmentation model, has an accuracy advantage, but the computational overhead of self-attention is not particularly friendly to embedded platforms like Jetson.

SegFormer is a lightweight Transformer segmentation framework proposed by NVIDIA Research in 2021. Its core innovation lies in replacing the traditional ViT with a hierarchical Mix Transformer (MiT) encoder and using an extremely simple All-MLP decoder. Unlike early vision Transformer approaches such as SETR, SegFormer progressively reduces sequence length in the encoder stage to control computation, striking a good balance between accuracy and efficiency. B0 through B5 represent six tiers from light to heavy; B2 is a mid-size variant with roughly 25M parameters, achieving over 46% mIoU on urban benchmarks like ADE20K. The YCOR (Yamaha-CMU Off-Road) dataset is designed specifically for unstructured off-road scenes, covering typical categories such as grass, dirt, shrubs, rocks, and water. It is one of the most widely used public benchmarks in off-road semantic segmentation, and its annotation granularity and category design directly determine the boundary of what a model understands as "traversability."

Core Pain Point: Dangerous "False Traversable" Predictions

From an engineering deployment perspective, the problems encountered in this project are more serious than simple frame rate deficiency:

• Nearby obstacles misclassified as traversable terrain — the error type with the greatest safety risk
• Beams, walls, and pillars occasionally recognized as pathways
• Random erroneous color patches appearing on open ground
• Only RGB input is currently used, while the stereo depth from the ZED 2i remains unused

For a navigation system, misclassifying an obstacle as a traversable region (false traversable) is fatal. The model should err on the side of caution and flag questionable areas as non-traversable, rather than letting a robot crash into a wall it mistook for a "road." A 57.4% mIoU may seem passable, but average metrics often mask fragile performance on critical categories — and it is precisely these rare but high-risk errors that determine whether a system can operate safely.

Improving Reliability: RGB-Depth Fusion Is the Key Breakthrough

To address misclassification, the most valuable improvement is to incorporate the depth information that is already available but unused. The ZED 2i can output dense depth maps natively, and the type of error where nearby obstacles are misclassified as traversable terrain is exactly what monocular RGB struggles most to solve — visually, a wall's texture may resemble the ground, but the depth geometry is entirely different.

Common RGB-D fusion strategies include:

• Early fusion: Concatenate depth as an additional channel with RGB before feeding into the network — simple to implement but requires adaptation of pretrained weights
• Mid-level fusion: Cross-fuse the two modalities at the feature level, typically using a dual-branch encoder structure
• Geometric post-processing: Use the depth map as a geometric constraint to filter out "traversable" regions in the segmentation output that violate physical laws

For high-risk scenarios like nearby obstacle misclassification, even using depth for simple post-processing validation (e.g., detecting abrupt height gradients) can significantly reduce the occurrence of false traversables — with virtually no added training cost.

Reducing Latency: From Quantization to Knowledge Distillation

Pushing frame rates from 20–30 FPS to 40–60 FPS requires a multi-pronged approach:

Model Compression and Quantization

TensorRT FP16 is already in use; the next step is to try INT8 quantization. INT8 can deliver significant speedups on Jetson AGX, but is sensitive to accuracy, requiring quantization-aware training (QAT) or a calibration dataset to keep accuracy loss under control. Given the project's extreme sensitivity to misclassification, it is recommended to evaluate per-category IoU for critical classes — not just overall mIoU — after quantization.

Knowledge Distillation

Using SegFormer-B2 as the teacher model and distilling into a lighter student network (such as Fast-SCNN or PIDNet-S) offers the potential to gain speed while retaining most of the accuracy. This is the classic approach to solving the "lightweight model lacks accuracy" problem — letting the smaller model learn from the soft-label knowledge of the larger one.

Architecture Selection

PIDNet was designed from the ground up for real-time semantic segmentation, and its three-branch structure strikes a reasonable balance between accuracy and speed. If the Transformer overhead of B2 proves truly difficult to compress, consider smaller variants like SegFormer-B0/B1, or invest more deeply in real-time architectures from the PIDNet family, using better training strategies and data augmentation to close the accuracy gap.

TensorRT is NVIDIA's high-performance deep learning inference optimization library, designed for deploying models on NVIDIA GPUs, including the Jetson embedded platform series. It dramatically reduces inference latency through operator fusion, memory optimization, and precision calibration. FP16 (half-precision floating point) is the most commonly used quantization level on Jetson AGX, delivering roughly 1.5–2× speedup over FP32 with minimal accuracy loss. INT8 (8-bit integer) can further double throughput on Tensor Cores, but requires a representative calibration dataset to profile the activation distribution of each layer, or the use of quantization-aware training (QAT) to simulate quantization error during training. Knowledge Distillation was systematically formalized by Hinton et al. in 2015. The core idea is to have a small network (student) fit the soft probability distributions output by a large network (teacher) rather than hard labels, thereby transferring the "dark knowledge" embedded in the larger model to the smaller one. In segmentation tasks, imitation losses on intermediate feature layers are typically added to improve effectiveness.

Training Strategy: Don't Just Chase mIoU

Random erroneous color patches and misclassification of specific objects (beams, pillars, columns) often reflect insufficient training data coverage or class imbalance. A few practical recommendations:

• Apply hard example mining or adjust loss weights for frequently misclassified categories
• Improve sample diversity of obstacles (walls, pillars, beams) in the training set
• Introduce temporal consistency constraints, leveraging continuity between video frames to suppress randomly flickering error patches
• Add safety-oriented metrics focused on the traversable/non-traversable binary classification during evaluation, rather than relying solely on multi-class mIoU

mIoU (mean Intersection over Union) is the most common evaluation metric for semantic segmentation, computed as the arithmetic mean of IoU across all categories. Its limitation is insensitivity to class imbalance — rare but high-risk categories (such as nearby obstacles) can have very low IoU with minimal impact on overall mIoU, easily masked by strong performance on frequent categories like large expanses of grass. In safety-critical navigation scenarios, more meaningful metrics include: Recall for the traversable/non-traversable binary classification, especially obstacle-class recall; and the False Traversable Rate — the proportion of pixels predicted as traversable but actually obstacles. Temporal consistency constraints are typically implemented via optical flow alignment losses between video frames or CRF (Conditional Random Field) post-processing, effectively suppressing the "flickering" noise common in single-frame inference and improving both perceived quality and the stability of downstream planning modules.

Closing Thoughts

This case is a microcosm of edge AI engineering in practice: the real challenge is never just about benchmark scores, but about finding a deployable balance among compute, latency, accuracy, and safety. For autonomous off-road navigation, fully utilizing the stereo depth information already on hand and conducting targeted optimization for high-risk misclassifications is often more valuable than blindly chasing higher mIoU or frame rates. Interested developers can explore the open-source implementations of SegFormer and PIDNet, as well as off-road datasets like YCOR, and run thorough benchmarks on real hardware.