Before Adding More Data, Check Whether Your Labeling Rules Are Stable

In computer vision (CV) projects, a common scenario plays out: model performance plateaus, and the team's first instinct is to "add more data" or "try a different model." These approaches sometimes work, but in many cases, the root cause is completely overlooked—your model is being asked to learn a rule that even humans haven't agreed upon.

This article is based on a practical observation shared on Reddit by a practitioner from Supervisely (a computer vision platform), exploring a severely underestimated issue: the stability of labeling rules often determines the model's upper bound more than data volume does.

Why Blindly Adding Data Amplifies Problems

Many teams default to "more data is better," but this assumption has an implicit prerequisite: the annotations are consistent.

In reality, the "correct answer" for many difficult samples is inherently ambiguous. For example:

• Partially visible objects: A car that's only half-visible, or an occluded pedestrian—should they be labeled or not?
• Uncertain boundaries: Where the object-background boundary is blurry, where exactly should the bounding box be drawn?
• Objects cropped by the frame: An object at the image edge with only a corner showing—does it count as an instance?

For these cases, two annotators can easily produce different yet both "justifiable" answers. When this inconsistency exists, adding more data only multiplies the chaos. The model receives contradictory supervision signals, and performance naturally struggles to improve—the more data you feed it, the greater the noise.

How Annotation Inconsistency Creates Chaos Inside the Model

To understand the severity of this problem, we need to return to the fundamentals of supervised learning. During training, the model updates its parameters by minimizing the loss function between predicted values and annotated values (ground truth). When samples of the same type have contradictory annotations, the loss function is essentially applying gradient forces in opposite directions—the model is being pulled in two different directions simultaneously, causing weight updates on these samples to cancel each other out. This is known in academia as the "label noise" problem. Research shows that even 5%-10% label noise can significantly decrease model accuracy on boundary samples, and deep neural networks, due to their powerful memorization capacity, will even attempt to memorize these contradictory annotations, leading to overfitting noise rather than learning genuine patterns. In other words, the model isn't "unable to learn"—it's exhausting its capacity on an impossible task.

A Low-Cost Annotation Consistency Check Method

The original author proposed a highly practical, nearly zero-cost verification workflow. The core idea is: diagnose the rules first, then scale up.

Step 1: Have Two People Independently Label Difficult Samples

Select 20–30 difficult images from your dataset (note: not random samples, but those with many edge cases and high ambiguity potential), and have two annotators complete the labeling independently.

Step 2: Review Disagreements, Not Just Agreement Rates

This is the key distinction. Most teams only care about the "consistency score" as a number—they're satisfied when they see 90%. But the truly valuable information is hidden in that 10% of disagreements.

Don't just look at what you agreed on—seriously study why you disagreed.

In annotation quality research, classic metrics for measuring annotator agreement include Cohen's Kappa coefficient (for two-person scenarios) and Fleiss' Kappa (for multi-person scenarios). Unlike simple percentage agreement rates, Kappa coefficients exclude random agreement, providing a more accurate consistency assessment. For object detection tasks, a common approach is calculating IoU (Intersection over Union) between different annotators' bounding boxes, typically using 0.5 or 0.75 as the threshold for judging whether two annotations "agree." But as mentioned earlier, looking at numbers alone can mask problems—behind an overall 90% agreement rate, specific categories or scenarios might have agreement rates as low as 50%, and these local issues are the true source of model performance bottlenecks.

Every recurring disagreement should be converted into a written rule, accompanied by one positive example and one negative example (each with a visualization). This transforms vague verbal conventions into executable, reproducible annotation standards.

Step 3: Retest with New Samples

After writing the rules, don't immediately start large-scale annotation. Instead, run the same two-person test on a batch of entirely new samples to verify whether the new rules truly eliminate disagreements. Only when the rules are proven stable is it worth investing resources to scale up data volume.

Auto-Labeling Needs This Same Logic

With the proliferation of auto-labeling and pre-annotation tools, many teams evaluate tools by inference speed. But the original author reminds us that the metrics for evaluating auto-labeling should be different:

• Number of missed objects (missed objects)
• Manual correction time per image (correction time per image)

And not just inference speed.

This point hits a nerve: if pre-annotation is blazing fast but every image still requires comprehensive manual review, that "speed" is meaningless. A high-speed but low-quality auto-labeling system might actually introduce new inconsistencies by tempting annotators to cut corners (directly accepting incorrect pre-annotations).

The Anchoring Effect Trap in Pre-Annotation

The auto-labeling workflow typically involves using pretrained models or existing models to perform initial inference on new data, generating candidate annotations for human review and correction. This "pre-annotation + human verification" approach is widely adopted in industry, but it introduces a hidden danger from cognitive psychology—the anchoring effect. When annotators see existing pre-annotation results, their judgment is unconsciously pulled by this "anchor," inclining them to accept rather than overturn it, even when the pre-annotation contains minor errors. This means low-quality pre-annotations aren't "neutral"—they systematically pull annotators' judgments toward the wrong direction, producing a subtle bias that's harder to detect and correct than starting from scratch. Therefore, truly valuable auto-labeling is the kind that genuinely reduces manual workload without sacrificing quality—its quality must be high enough that "accepting the default" is the correct choice in most cases.

Practical Implications for CV Project Teams

Although this observation comes from a platform practitioner, the author emphasizes it's a universal lesson independent of any specific platform. Here are several implications for teams:

First, when performance plateaus, suspect the annotations before the model. Before adding data or switching architectures, spend half a day running a two-person consistency test. The cost is minimal; the payoff can be enormous.

Second, annotation rules should be documented, visualized assets. Verbal agreements and "going by feel" collapse as teams grow. Writing every edge case into rules with positive and negative examples is the most easily overlooked infrastructure in data engineering.

As a reference from well-known datasets: the COCO dataset's annotation guidelines span dozens of pages, with explicit definitions for edge cases of every object category; industrial-grade annotation teams typically maintain an "edge case library," archiving every discovered ambiguity as a rule. These rules usually follow a format including: trigger condition description, correct annotation example (positive case), common incorrect annotation example (negative case), and the business logic explanation behind the rule. As teams scale or annotators turn over, this document becomes the core infrastructure for ensuring data consistency—its importance is comparable to coding standards in a codebase.

Third, evaluate tools with the right metrics. Whether for manual or auto-annotation, evaluation criteria should center on "final data quality" and "actual labor saved," not superficial speed or agreement rate numbers.

Conclusion

Data quality is the ceiling for model performance, and the stability of labeling rules is the ceiling for data quality. Before pursuing larger datasets or stronger models, take a step back and ask yourself: Do we truly know what a "correct" annotation looks like?

If two people can't agree on the same image, no amount of data will save your model. Which annotation edge cases in your dataset give you the biggest headaches? That might be exactly where your next performance breakthrough lies.