The CoreML Model Is Fine, Your Photos Are Too Big: The Image Pipeline Trap in On-Device AI

A mundane computer vision failure: the model works as specified, but the image pipeline makes the task impossible.

A developer shared his experience: the app needed to count livestock on-device using a small YOLO/CoreML model — no network, no account, no server-side fallback. That constraint is useful, but it means the mobile pipeline has to honestly confront what the model actually sees.

250 sheep, and the model counts only 14

The initial code path looked reasonable:

let request = VNCoreMLRequest(model: model)
request.imageCropAndScaleOption = .scaleFit
handler.perform([request])

One image in, detections out.

Then he tested with a real ranch photo — roughly 250 sheep in the frame. The count came back as 14.

This isn't a model problem. A 12-megapixel photo got squeezed into the model's input. By the time the flock reaches the network, distant sheep are only a few pixels each. The detector has nothing clever left to recover.

Tiled inference: don't change the model, change the input size

The fix is boring, but far better than swapping models:

1. Run the full image first
2. Run overlapping tiles on the same image
3. Map tile-relative detection boxes back to image coordinates
4. Discard boxes cut off by internal tile edges
5. Merge the remaining detections with class-agnostic NMS

The key point is that tiling changes the object size the model sees, without changing the model itself.

let regions = [CGRect(x: 0, y: 0, width: 1, height: 1)]
    + Self.tiles(aspect: aspect, across: 3, overlap: 0.25)

for region in regions {
    request.regionOfInterest = region
}

On a 4:3 photo, this becomes 12 tiles plus one full-frame inference. More inferences, but still on-device, and still under a second for a static photo.

The same ranch photo went from 14 detections to 173. Not perfect, but suddenly useful. A close-up photo of hens stayed stable, which matters just as much: the fix improved recall where scale was the problem, without inflating counts in easy-to-count scenes.

What is regionOfInterest

VNCoreMLRequest's regionOfInterest (ROI) is a property provided by the Vision framework. Its value is a CGRect in normalized coordinates (0–1); the default .zero means the entire image. When set, Vision first crops that rectangular region from the original image, then scales it according to imageCropAndScaleOption and feeds it to the model. This means the actual pixel size of each inference input may differ, but the model itself (input tensor shape, weights) is completely unchanged. Precisely because ROI only changes "the pixels fed to the model," tiled inference becomes a cheap optimization that doesn't touch the model architecture. Note that coordinate normalization is based on the original image orientation — rotated or mirrored images must be normalized first, or the ROI will crop the wrong location.

The trade-offs of imageCropAndScaleOption

imageCropAndScaleOption determines how Vision fits an image to the model's required input size. Common values are .centerCrop, .scaleFit, .scaleFill, and .scaleFitRotate90CCW. .centerCrop preserves aspect ratio but crops edges, easily losing targets at the edges of wide photos; .scaleFit preserves the entire frame but pads gray bars on the short side, effectively shrinking objects; .scaleFill stretches and distorts. When the article's 12-megapixel photo is scaled down to the model input (typically 416×416 or 640×640), a single distant sheep may occupy only a few pixels. This is a classic "extreme-scale downsampling" problem, not a lack of model capability. Choosing .scaleFit combined with tiling lets objects within each tile occupy more pixels, essentially compensating for scale loss at the input layer.

The merge step is the hidden trap

The non-obvious part is the merge step.

Default IoU-only NMS isn't enough with tiling. A partial head box may sit inside a whole-animal box with low IoU, so you also need intersection-over-smaller merging. And make the merge class-agnostic, because the same animal may come back with two labels across different inferences. Count animals, not label arguments.

The lesson is simple: before blaming the model, check what pixels you actually gave it.

For on-device AI, the model is often the least flexible part of the system. The image pipeline around it is where the cheap optimizations hide.

IoU vs. intersection-over-smaller (IoS)

NMS typically uses IoU (intersection over union) to measure the overlap of two boxes: intersection area divided by union area. But when one box is completely contained within another — for example, in a tiled scenario where a "partial head" box falls inside a "whole animal" box — the intersection area is large, but the union equals the larger box, so IoU comes out noticeably small, and boxes that should be merged get kept. Intersection-over-smaller (IoS) instead divides the intersection area by the area of the smaller box. As long as the smaller box falls mostly inside the larger one, IoS approaches 1, correctly triggering a merge. In tiled inference, the same target may appear as a local box and a global box in adjacent tiles, so IoS is a complementary criterion designed precisely for this kind of "containment overlap." Combined with class-agnostic merging, it prevents the same animal from being double-counted due to inconsistent labels or scales.

Class-agnostic NMS

Standard NMS usually runs per class, suppressing only when two boxes share the same class and their IoU exceeds the threshold. But in tiled inference, the same target may be assigned different labels by the model across different tiles or contexts (e.g., "sheep" and "animal"), and per-class filtering will keep the same entity as multiple boxes. Class-agnostic NMS ignores class, applies overlap suppression uniformly across all candidate boxes, and then decides which label to keep. For counting-focused scenarios, this strategy is critical: the thing you're counting is a physical entity, not a classification debate. The cost is that it may suppress some genuine targets that have different classes but spatially overlap, so thresholds and whether to retain class information should be weighed according to the task.

One-line Summary

When on-device visual counting is off, don't swap the model first — tiled inference and the right merge strategy can often take recall from 14 to 173.

Paragraph Summary

The article documents a typical on-device computer vision failure: a developer used a small YOLO/CoreML model to count livestock on-device, but a 12-megapixel photo of roughly 250 sheep yielded only 14 detections. The problem wasn't the model itself, but that the full high-resolution photo was scaled straight down to the model input size, leaving distant small targets with only a few pixels — nothing a detector can recover. The fix was boring but effective tiled inference: keep the full-frame inference, stack 3×3 tiles with 25% overlap, map tile coordinates back to the original image, discard boxes cut off by tile edges, and merge results with class-agnostic NMS. Detections on the same photo went from 14 to 173, without inflating counts in simple close-up scenes. The article stresses that the merge step is the hidden trap — in tiled scenarios IoU alone isn't enough; you also need intersection-over-smaller and class-agnostic merging. The core lesson: before blaming the model, check what pixels you actually fed it. In on-device AI, the image pipeline is often the cheapest and most effective place to optimize.