Pothole Detection Model Misclassifying Roadsides? A Practical Guide to Object Detection Optimization

A Common Beginner's Dilemma

Among introductory computer vision projects, pothole detection is both practical and challenging. Recently, a beginner shared their frustration on Reddit: their trained pothole detection model frequently misclassified roadsides as potholes during the first few seconds of video. While detections improved later in the clip, the false positives seriously undermined the model's practical utility.

This developer used approximately 4,100 images for training and plans to eventually deploy the model on a drone for road inspection. They raised a series of highly representative questions: How can the model distinguish between roadsides and potholes? Should negative samples be added? How should small targets be handled from a drone's perspective? And is segmentation more appropriate than detection?

These questions span the entire pipeline of an object detection project—from data to deployment—and deserve a systematic walkthrough.

The Root Cause of Roadside Misclassification: False Positive Analysis

When a model mistakes roadsides for potholes, it's fundamentally a False Positive (FP) problem. False positives are one of the core concepts in machine learning evaluation—in the context of object detection, they refer to the model incorrectly identifying background regions or non-target objects as the target class. The counterpart is False Negative (missed detections), along with True Positive (correct detections) and True Negative (correctly ignored background). These four metrics together form the confusion matrix, from which Precision, Recall, and F1 score are derived. In safety-related applications like pothole detection, too many false positives lead to unnecessary repair dispatches that waste resources, while too many false negatives miss real road hazards. The right balance between precision and recall must be found based on the actual deployment scenario.

Three underlying factors typically contribute to this issue:

Lack of Background Negative Samples in Training Data

If nearly all images in the training set contain potholes, the model tends to output "pothole" for any region that looks rough or has unusual coloring. Roadsides, with their complex textures and varying shadows, easily trigger such misclassifications.

Regarding the poster's question about whether to add a "non-pothole" class, the answer is yes—but the approach matters:

• For detection models (like the YOLO family): You don't need a separate "non-pothole" class. Instead, add background negative sample images with no annotations. YOLO (You Only Look Once) is one of the most influential single-stage detection frameworks in object detection, first introduced by Joseph Redmon in 2016. Unlike traditional two-stage detectors (such as Faster R-CNN, which first generates region proposals then classifies them), YOLO reformulates detection as a single regression problem, predicting bounding box locations and class probabilities in a single forward pass. As of 2024, YOLO has evolved to YOLOv8/YOLOv11 (maintained by Ultralytics), achieving excellent balance between detection accuracy and inference speed. In YOLO's training pipeline, unannotated background images teach the model that even when certain regions resemble targets in texture or color, they should not be detected—all prediction boxes the model generates on these images are treated as false positives and suppressed through backpropagation of the loss function. These images should include roadsides, manhole covers, shadows, tree shadows, water stains, and other confusable objects, but with no bounding box annotations. The model learns that "these regions should not be boxed."
• For classification models: A "background" or "non-pothole" class is indeed needed.

Regarding the number of negative samples, community experience generally suggests that negatives should comprise 10%–20% of the total dataset. With 4,100 positive samples, adding 400–800 targeted roadside and road surface background negatives is a good starting point. The key isn't quantity but ensuring that these negative samples cover scenarios where the model actually makes mistakes.

Lack of Road Structure Context Information

Another idea the poster mentioned—"teaching the model to recognize road edges so it doesn't mistake roadsides for potholes"—is a more advanced but viable approach. This essentially introduces spatial constraints or multi-class detection:

You can simultaneously annotate "road (drivable surface)" and "pothole" as two classes, letting the model understand that potholes should only appear within road areas. During post-processing, any detection boxes falling outside the road region can be filtered out. While this increases annotation costs, it can significantly reduce false positives in boundary areas.

Data Augmentation and Annotation Quality Optimization

Beyond adding negative samples, improving model robustness requires attention to data quality itself.

Check Annotation Consistency

In beginner projects, false positives often stem from inconsistent annotations. For instance, some images include cracks adjacent to potholes within the bounding box while others only box the center. It's advisable to re-audit the dataset to ensure consistent boundary definitions for potholes, preventing the model from learning ambiguous criteria.

Targeted Data Augmentation Strategies

For misclassifications caused by lighting and shadows, introduce brightness, contrast, and hue jittering augmentations to make the model insensitive to illumination changes. Additionally, adding random occlusion and blur augmentations can improve generalization in real road scenarios.

Drone Deployment: Transfer Learning and Small Object Detection

The poster plans to use drones to capture road imagery and has identified a key issue: the drone's flight altitude makes potholes appear very small in the frame, which differs drastically from ground-level or vehicle-mounted camera perspectives. This discrepancy between training data distribution and deployment data distribution is called Domain Gap—it can arise from shooting angle (ground vs. aerial top-down), lighting conditions (sunny vs. overcast vs. nighttime), geographic regions (different road materials and pothole morphologies across countries), and camera parameters (focal length, resolution, distortion). When the domain gap is large, features learned on the source domain may not transfer effectively to the target domain, causing significant performance degradation.

Prefer Transfer Learning Over Training From Scratch

Training from scratch is almost never recommended unless you have massive data and abundant compute. The correct path is:

1. Use a model pre-trained on a large-scale dataset (like COCO) as your foundation;
2. Fine-tune on your pothole dataset;
3. Once sufficient real drone-perspective data is collected, perform a second round of targeted fine-tuning.

Transfer learning is one of the most important practical paradigms in deep learning. Its core idea is that models pre-trained on large-scale datasets have already learned general visual feature representations (such as edges, textures, shapes, and other low-level features) that can transfer to new downstream tasks. COCO (Common Objects in Context) contains 80 categories, over 330,000 images, and 1.5 million object instances, making it the most commonly used pre-training benchmark in object detection. In transfer learning, shallow backbone layers are typically frozen or updated with low learning rates, while the detection head and deeper layers are trained with higher learning rates. This strategy allows models to converge quickly and achieve good performance even with just a few thousand domain-specific images. For relatively niche tasks like pothole detection, transfer learning is virtually a mandatory choice.

Optimization Methods for Small Object Detection from Drones

Potholes viewed from high-altitude drone perspectives are a classic small object detection challenge. Here are several approaches:

• Increase input resolution: Raising the input size of models like YOLO from 640 to 1280 or higher can significantly improve small object recall (at the cost of slower inference);
• Tiling / SAHI (Slicing Aided Hyper Inference): Splitting large images into multiple overlapping sub-patches for separate inference then merging results is a classic technique for handling small objects in drone scenarios. SAHI was proposed by Fatih Cagatay Akyon et al. in 2022 as an inference-time enhancement framework. Its core principle is to slice a high-resolution image into multiple overlapping patches of fixed size (e.g., 640×640), perform object detection inference on each patch separately, map all detection results back to the original image coordinate system, and finally merge duplicate detections in overlapping regions through Non-Maximum Suppression (NMS). This method requires no model retraining—it's used only during inference and can significantly improve small object recall, with the trade-off being that inference time scales proportionally with the number of patches;
• Adjust anchor settings: Re-cluster anchor box sizes specifically for small targets.

Additionally, training data should ideally be captured from perspectives close to the deployment altitude, minimizing the domain gap between training and inference.

Detection or Segmentation? How to Choose

The final question is: should you use object detection or semantic/instance segmentation? The poster is interested in "whether pixel-level area calculation of potholes is possible."

Image segmentation in computer vision exists at multiple levels. Semantic Segmentation assigns a class label to every pixel in an image but doesn't distinguish between different instances of the same class—for example, two adjacent potholes would be labeled as one "pothole" region. Instance Segmentation goes further by not only identifying classes but distinguishing individual objects, separately marking each independent pothole. Representative models include DeepLab and SegFormer for semantic segmentation, and Mask R-CNN and YOLOv8-seg for instance segmentation.

The choice depends on project objectives:

• If you only need to locate and count potholes, detection (bounding boxes) is sufficient—it's simpler to implement, faster for inference, and cheaper to annotate.
• If you need to estimate pothole area, shape, or severity (which is highly valuable for road maintenance assessment), instance segmentation is more appropriate. Segmentation provides pixel-level pothole contours, and combined with the drone's GSD (Ground Sampling Distance—the actual ground size each pixel represents, determined by flight altitude, camera focal length, and sensor size), you can calculate real physical areas.

For beginners, the recommendation is to get detection working solidly first, resolve the current false positive issues, then gradually transition to segmentation. Segmentation annotation requires tracing pixel-level contours (polygon annotation), typically costing 5-10x more than bounding box annotation. Jumping in prematurely can easily lead to data bottlenecks.

Summary: A Roadmap from False Positive Fixes to Full Deployment

This beginner's list of questions actually outlines the complete evolution path of a practical computer vision project:

1. Fix false positives first: Add targeted background negative samples and check annotation consistency;
2. Introduce contextual constraints: Use multi-class detection to help the model understand spatial positioning of potholes;
3. Embrace transfer learning: Fine-tune from pre-trained models for drone deployment rather than training from scratch;
4. Optimize for small objects: Increase resolution or adopt tiling inference;
5. Choose segmentation as needed: Adopt segmentation only when area estimation is required.

Starting from a specific bug like "the model mistakes roadsides for potholes" and progressively diving into the full pipeline of data, modeling, and deployment thinking—this is the essential path for growing computer vision engineering skills.

Key Takeaways