A Beginner's Guide to Computer Vision: From Pixels to YOLO and Image Segmentation

How Do Computers "See" Images?

A developer shared his computer vision learning journey, starting from the most basic question: how does a computer identify people, cars, or objects in a picture?

The answer lies in numbers. A digital image is made up of pixels, each containing color or brightness information at a specific position. RGB is the most common color representation—red is (255, 0, 0), green is (0, 255, 0), and blue is (0, 0, 255). A 200×300 color image contains three channels, totaling 180,000 color values.

This means computers don't "see" photos the way humans do. They process numerical information that represents the image. Understanding this is the first threshold to cross when entering the field of computer vision.

Computer vision learning journey

Image Processing Basics and the Intuition Behind Convolution

Once you grasp the concept of pixels, the next step is learning to transform images with Python. Common operations include:

Grayscale conversion: Converting a color image to grayscale simplifies certain processing tasks but loses color information
Blurring: Reducing image sharpness can decrease noise, but excessive blurring can obscure important details
Sharpening: Making edges and details clearer, but over-sharpening degrades image quality
Edge detection: Identifying boundaries where brightness changes, which often correspond to object contours

The Sobel operator uses small filters to detect brightness changes, which leads to the core idea of convolution. Filters can be applied to image pixels to identify patterns, and convolutional neural networks (CNNs) can automatically learn useful filters from data.

Convolution is one of the most fundamental operations in computer vision. It works by taking a small matrix called a "convolution kernel" or "filter" (for example, a 3×3 grid of values) and sliding it across the image pixel by pixel, each time multiplying the kernel values by the corresponding pixel values and summing them to produce a new pixel in the output image. Different kernels produce different effects—detecting horizontal or vertical edges, blurring, and sharpening are all essentially different convolution kernels at work. The Sobel operator is a classic pair of convolution kernels used to approximate image gradients, responding to horizontal and vertical brightness changes respectively, which is why it can highlight object contours.

Traditional methods require manually designing these kernels, but the key breakthrough of convolutional neural networks (CNNs) is that they treat the values inside the kernel as learnable parameters, automatically optimizing them through backpropagation and large amounts of data, thereby learning feature extractors that humans would struggle to design by hand. CNNs typically stack multiple convolution layers—shallow layers learn low-level features like edges and textures, while deeper layers combine them into object parts and even complete objects. This is the fundamental reason CNNs far outperform previous approaches on image tasks.

Image Classification: How Models Determine "What Is This"

The task of image classification is to predict what content an image contains—for example, determining whether it's a dog, a car, or a computer screen.

When this learner used a pretrained model to analyze an image, the model returned multiple candidate labels with scores: Website ranked first with an 87.5% score, Menu at 8.4%, Screen at 2.2%, Television at 0.7%, and Brass at 0.4%.

This experiment revealed a key insight: classification models provide multiple possibilities ranked by score, and the highest score doesn't guarantee a correct prediction. Image classification only predicts "what's there" without indicating where objects are located in the image—this is precisely the core difference from object detection.

How Object Detection and YOLO Work

Object detection not only identifies objects but also locates them. The model's output typically contains three elements:

Label: The name of the detected object
Bounding box: The rectangular region marking the object's location
Confidence score: The confidence level associated with the prediction

Multiple people in a photo can be individually boxed, which classification models cannot do.

YOLO (You Only Look Once) is a widely used family of models for object detection. Its pipeline roughly goes: input image → process visual features → predict possible objects → filter results → output bounding boxes and labels. But detection models can produce duplicate or incorrect predictions, and results need human verification.

Bounding boxes are the standard way to describe object locations in object detection, typically represented by four numbers: the x and y coordinates of the top-left corner, plus the width and height of the box (sometimes using two sets of coordinates for top-left and bottom-right corners). It assumes an object can be roughly enclosed by a rectangle parallel to the image edges, so it fits elongated or tilted objects poorly.

Confidence scores reflect the model's confidence that a prediction "both contains an object and is accurately localized," generally consisting of two parts: the probability that an object exists within the box, and the degree of overlap between the predicted box and the ground truth box. A commonly used metric for evaluating localization quality is IoU (Intersection over Union), which is the intersection area of the predicted and ground truth boxes divided by their union area, ranging from 0 to 1. The closer to 1, the more accurate the box.

YOLO's design philosophy is "look only once": unlike earlier two-stage methods that first generate a large number of candidate regions and then classify them one by one, it treats detection as a single regression problem, simultaneously predicting classes and bounding boxes at all positions in a single forward pass. This makes it extremely fast and suitable for real-time video detection. Precisely because a single image can output hundreds or thousands of candidate boxes at once, non-maximum suppression (NMS) is used in post-processing to remove duplicate boxes for the same object, keeping only the one with the highest score.

Confidence Thresholds: A Double-Edged Sword

The confidence threshold determines which predictions the model retains. Setting the threshold to 0.10 means using 10% as the cutoff line.

A low threshold retains more predictions, including incorrect ones; a high threshold filters out low-scoring predictions but may also mistakenly remove real objects.

In experiments, the learner observed the model outputting labels like cake, toothbrush, and scissors, but these labels didn't match the screenshot content. This directly demonstrates that confidence scores are not a guarantee of correctness. To evaluate model performance, predictions must be checked against the image itself.

When evaluating detection models, Precision and Recall are two fundamental metrics. Precision refers to how many of the model's predictions are correct, while recall refers to how many of the real objects were successfully found. The threshold is precisely the knob that balances the two: lowering the threshold increases recall but decreases precision; raising the threshold increases precision but increases missed detections.

Since the threshold affects both directions simultaneously, the industry commonly uses mAP (mean Average Precision) to comprehensively measure model performance. It works by: plotting precision-recall curves under different IoU criteria, calculating the area under the curve (i.e., Average Precision, AP), and then averaging across all categories. The higher the mAP, the better the model performs overall in balancing "finding them all" and "finding them accurately." Understanding this makes it clear why a single confidence score alone is insufficient to judge a model's quality—thresholds must be set based on the specific task's preference for precision versus recall.

Image Segmentation: A Finer Understanding Than Bounding Boxes

Image segmentation is another category of computer vision tasks, identifying regions and pixels belonging to specific classes or objects. Three main types:

Semantic segmentation: Assigning a class to each pixel, such as road, building, pedestrian
Instance segmentation: Distinguishing different individuals within the same class
Panoptic segmentation: Combining semantic and instance segmentation

Unlike object detection which uses bounding boxes, segmentation provides a finer representation of object shapes. The existence of segmentation shows that computer vision goes far beyond just classification and detection.

How Image Changes Affect Model Predictions

Changing the image itself changes the model's judgment. This includes dimming the image, rotating it up or down, reducing resolution, applying blur, and other operations.

Blur can obscure important details, and low resolution can lose information about small objects. Re-upscaling an image cannot recover details already lost when resolution was reduced.

Such experiments need careful design—to understand the effect of a certain transformation, you must use appropriate images and consistent settings for comparison.

Why Models Make Mistakes

Code running successfully doesn't mean the results are correct. Common reasons models make errors include:

Insufficient lighting or poor image quality
Objects being too small or partially occluded
Objects with similar appearances
Limitations of the model itself
Distribution differences between training images and analyzed images

These factors remind us: we cannot rely solely on model output—we need to verify predictions and test their performance.

Transfer Learning: Standing on the Shoulders of Existing Models

The idea of transfer learning is to leverage already-trained models and adapt them to new tasks. Instead of training from scratch, you start from a pretrained model and fine-tune it with appropriate data.

A typical scenario: using labeled local crop images to adapt a pretrained model to recognize specific crop diseases. The quality and relevance of training data are crucial, and the model also needs to be tested on images it hasn't seen during training to verify generalization ability. This is especially important for developing computer vision applications in different environments.

Transfer learning works because the features learned by deep networks are transferable. In models pretrained on large-scale datasets like ImageNet, the shallow and middle layers have already mastered general visual features such as edges, textures, and shapes, which are useful for the vast majority of image tasks. Only the higher layers near the output are strongly correlated with specific classes. Therefore, there are typically two strategies for fine-tuning: one is feature extraction, freezing all convolutional layers of the pretrained model and only retraining the final classification layer; the other is full model fine-tuning, updating all layers with a smaller learning rate to adapt features to the new domain.

There are several key points in practice: the smaller the new task's dataset, the more layers should be frozen to prevent overfitting; the learning rate should be much smaller than when training from scratch to avoid destroying well-learned weights; if the new data's distribution differs greatly from the pretraining data (for example, transferring from natural images to medical imaging), more layers often need to be fine-tuned. Testing generalization ability must rely on a test set never seen during training; otherwise, the model may have simply memorized training samples and will perform poorly in real scenarios.

Real-World Applications of Computer Vision

The learner listed several application directions in the article:

Agriculture: Identifying plant diseases, monitoring crop growth
Healthcare: Assisting medical image analysis under professional supervision
Transportation: Detecting vehicles, pedestrians, and road signs
Environment: Monitoring vegetation, wildlife, and land use changes
Accessibility: Helping visually impaired people understand surrounding objects and text

Each application needs careful testing in the target environment to ensure reliable operation.

Key Takeaways and Next Steps

The learner's summary covers the key insights of computer vision: images are represented by pixels and numbers; image processing techniques change image appearance; classification predicts content; detection identifies locations; YOLO enables efficient detection; confidence thresholds affect which predictions are retained; segmentation provides pixel-level information; image quality changes affect predictions; models make mistakes; transfer learning can adapt pretrained models.

He plans to continue learning about convolutional neural networks, model training, and evaluation metrics like precision, recall, and mAP, while improving Python skills and dataset preparation abilities.

From pixels to segmentation, each topic forms a different piece of the computer vision puzzle. Even as a beginner, understanding how models make predictions—and why predictions aren't always correct—is already significant progress.

One-line Summary

From pixels to segmentation, a developer breaks down the core concepts and common pitfalls of computer vision.

Paragraph Summary

A developer shared his computer vision learning journey: computers don't "see" images the way humans do—they process digital information made up of pixels and RGB values, and a single 200×300 color image contains 180,000 color values. The article walks through image processing operations like grayscale conversion, blurring, sharpening, and edge detection along with the intuition behind convolution, then introduces the differences between three task types—image classification, object detection, and image segmentation. Classification only predicts content, detection additionally outputs bounding boxes and confidence scores, and segmentation provides pixel-level understanding. Through experiments, the author found that high-scoring labels from models aren't always correct, confidence thresholds are a double-edged sword, and models make mistakes due to lighting, occlusion, distribution differences, and other reasons. Finally, the article discusses how transfer learning can leverage pretrained models to adapt to new tasks, and lists application scenarios in agriculture, healthcare, transportation, environment, and accessibility.