Removing the Safety Net Made It Faster and Better: An Engineering Decision Log on Video Inpainting Optimization

This is an engineering decision log, not a mechanism tutorial. At its core, it tells a counterintuitive story: the ClearPix team deliberately removed a safety mechanism that was already live and working, replaced it with a simpler design, and ended up not with regression — but with a 1.5dB quality gain in masked-region PSNR over the old baseline.

If there's one takeaway from this article, it's this discipline: define your stopping criteria before running experiments, and make sure your test set includes the worst-case footage from your most severe historical bug.

Starting From a Working System

Some context first. ClearPix's browser-based video watermark removal tool performs per-frame inpainting, but most frames near a static watermark change very little. So the team built a "motion gate" to skip inference and reuse previous results.

Version 1 (v1) of the gate compared each frame to its neighbor and reused the previous frame's output. This design failed catastrophically once: on footage with a slow global background drift, the per-frame difference was always tiny, so the gate would always skip, and the inpainted region would freeze into a visually obvious rectangular block. This incident sent end-to-end masked-region PSNR plummeting from 38.7dB to 16.7dB — a 22dB loss.

After the incident, the team built a dedicated regression clip: a synthetic segment with continuously slow background drift, specifically constructed so that "no future change could silently reintroduce chained freezing."

The fix was a "cumulative drift budget + luminance delta compensation" scheme: accumulated error from skipped frames was tracked, and once the budget was exhausted, inference was forced; meanwhile, reused pixels were fine-tuned based on scene-measured exposure offsets. It worked — the frozen-rectangle bug stopped recurring, quality held at 38.44dB (vs. 38.74dB with no skipping), and the safety net shipped to production.

So the starting point of this story isn't a broken system — it's a working system with one extra layer of indirection.

---

PSNR (Peak Signal-to-Noise Ratio) is the standard metric for measuring image reconstruction quality, expressed in decibels (dB) — higher values indicate closer fidelity to ground truth. It's computed from mean squared error (MSE): PSNR = 10 · log₁₀(MAX² / MSE), where MAX is typically 255 for 8-bit images. Because it's a logarithmic scale, a 3dB difference means error power is halved, which corresponds to a perceptible quality difference; the 22dB drop (38.7 → 16.7) means the reconstructed image in the masked region is near total distortion, with clearly visible artifacts. All quality comparisons in this article use "masked-region PSNR," computed only within the watermark coverage area rather than the full frame — this is appropriate because the skip-frame logic only affects inpainting output, and the high-quality background would severely dilute the true degradation signal in a full-frame metric.

The Anchor Idea: Why Compare to a Neighbor?

At a design review, someone asked the question that started everything: why does each frame compare to its neighbor?

Think about what skip-frame error actually is. When we skip a frame, we're reusing the output of "the last frame that actually ran inference" — call it the anchor. The error we're incurring is exactly "how different is the current frame from the anchor."

The drift budget only approximates this quantity: it accumulates step-wise differences and hopes these steps sum correctly. Comparing directly to the anchor measures this quantity directly: by definition, the skip-frame error is bounded by the threshold, with no accumulator, no proxy, and no edge cases where slow drift might slip through undetected.

The theory also predicted no speed loss. Accumulating noisy step-wise differences only "overestimates" relative to direct comparison (oscillations and noise accumulate in summation but cancel in direct differencing), so skip rates would be equal or better. Fewer state variables, stronger guarantees, equal or better performance.

Precisely because this kind of idea is elegant, it's also the most likely to blow up in practice — so the team wrote down the rules first.

---

Temporal EMA (Exponential Moving Average) is a common temporal smoothing technique in video processing: blending the current frame's inpainting output with the previous frame's output at a fixed weight ratio to reduce temporal flickering. Its recurrence is output_t = α · inpaint_t + (1−α) · output_{t−1}, where α is typically 0.3–0.7. The key point: EMA's "history term" always refers to the previous frame, not the anchor frame. So when motion-compensating the history term, you must use the displacement delta from "current frame → previous frame," not "current frame → anchor frame." The second bug described in this article is exactly this semantic mismatch: using the anchor delta for anchor reuse is correct, but feeding that same delta into EMA's history term means applying the wrong offset to pixels on an entirely different temporal reference — causing systematic overshoot rather than correction.

PSNR Regression Testing and Pre-Committed Stopping Criteria

The team's rule for pipeline changes: a change must pass end-to-end quality gates before shipping, no exceptions.

For this experiment, they locked in the gate before running anything: if masked-region PSNR on any end-to-end clip fell below 36dB, the refactor would be aborted — no patching, no negotiating. This number was chosen while the team was still neutral about the outcome. Setting a stopping criterion after seeing results is the canonical path by which regressions get rationalized and sneak into production.

The gate itself deserves attention, because this is where the real protagonist of this article makes its second appearance. The end-to-end suite included:

• The synthetic gradual-drift clip born from the 16.7dB chained-freezing incident — the worst case for any reuse-based optimization, where the background is never still but changes slowly enough to fool naive motion detection;
• A four-segment subtitle clip with continuously changing masked content.

One attacks "stale reuse," the other attacks "insufficient reuse." Whatever you change, one of them will find the problem.

---

A pre-committed stopping criterion is the core mechanism in experimental design for preventing outcome bias, analogous to pre-registration in A/B testing and statistics. Its psychological basis: once researchers see partial positive results, the brain automatically searches for reasons why negative results don't matter. This post-hoc rationalization is extremely common in engineering — typically expressed as "this test case wasn't representative," "we'll optimize it separately later," or "the overall average still improved." In ML model evaluation and A/B experiments, this is structurally identical to p-hacking (repeatedly adjusting evaluation criteria until results look good). The ClearPix team's approach was to define stopping criteria per-clip rather than on aggregate means, and to lock them in before the experiment — so that even if results split across clips, the decision process requires no judgment calls, only executing the agreed-upon contract.

First Run: Pass, Pass, Pass, Stop

After the anchor rewrite shipped internally, the first full run produced a "split" result:

• Subtitle clips: all four segments improved by +3–4dB (37.3 / 32.3 / 28.7 / 30.1dB). Direct comparison fully delivered on its theoretical promise for fast-changing content.
• Gradual drift clip: masked-region PSNR dropped from 38.30 to 34.69dB — below the 36dB stopping line.

This is exactly what the discipline exists for. The aggregate picture looked fine — even better. It would have been easy to average the results, declare victory on subtitles, and "tune the drift case later." The team didn't do that. They stopped as agreed, and treated 34.69 as a bug report written by the experiment.

There was also no rollback: the stopping line meant "stop and understand," not "stop and revert."

Error Accumulation, Round Two

Root cause analysis uncovered something genuinely new — a failure mode that the chained design had been hiding all along.

The gradient clip's background doesn't drift uniformly. Its exposure offset varies with vertical position (the clip was synthesized with roughly sin(y/60 + t·4) and cos(y/80 + t·3) components), while the delta compensation only takes a single per-channel mean across the entire region. A mean cannot cancel a gradient.

In the old chained design, a reused pixel was always exactly one frame old, so uncompensated residuals remained small and roughly constant. In the anchor design, a single reused output might be two or three frames away from its anchor, and residuals grow with anchor age. The refactor didn't create the compensation error — it amplified it by extending reuse lifetime. And that's the same mechanism responsible for making everything else better.

A second, lower-level bug was also found in the same area. Temporal EMA blends the current output with the previous frame's output, so its history term needs compensation relative to the previous frame. But in the rewrite, the team was feeding it the "to-anchor" delta — causing systematic overshoot on every blended pixel, because the compensation was answering a completely different question than the blending was asking.

---

The article mentions the background was synthesized with sin(y/60 + t·4) and cos(y/80 + t·3) spatial gradient components, meaning the luminance offset at different vertical positions within the same frame is different — the exposure change is a gradient field across space, not a uniform offset. Taking a single per-channel mean across the entire masked region is equivalent to fitting a constant to a position-varying function; the residual is spatially asymmetric, cannot cancel out, and persists as structured error. This "mean compensation meets spatial gradient" failure is common in video stabilization, exposure alignment, optical flow estimation, and similar domains. A more precise solution would use tile-wise or optical-flow-based per-pixel compensation. But the team's chosen fix was more pragmatic: lowering the skip threshold ceiling to force more frequent re-anchoring, keeping anchor age short enough that even coarse compensation prevents residuals from accumulating to visible levels.

Two Small Fixes, No Rollback

Both fixes were small and stayed within the new design:

• Threshold ceiling 8 → 5. The skip threshold is self-calibrating (median of first five frame differences × 3, clamped to [2, 5]/255). Lowering the ceiling forces more frequent re-anchoring, directly constraining anchor age — attacking the residual accumulation mechanism itself, not its symptoms.
• Split the delta. The compositing step reuses the anchor, so it uses the "to-anchor" delta; EMA blends with the previous frame, so it uses the "to-neighbor" delta. Each has its own reasoning chain, with no shared state pretending two different temporal references are the same thing.

The calibration logic itself was untouched — only the clamp ceiling and the delta "routing" changed.

Results: Beating the Baseline

The final anchor version, under the same gate:

• Gradual drift clip: masked-region PSNR reached 39.84dB — 1.5dB above the chained baseline, not just recovery. With delta semantics fixed, direct comparison plus precise "to-anchor" compensation proved strictly superior to the approximation the team had been defending all along.
• Subtitles: the +3–4dB gains held, with 67% of frames skipping inference.
• Speed: 58% of frames skipped on watermark footage, matching the budget version (as predicted), while the entire drift-budget code path was deleted.

The safety net that was removed was never missed. What actually worked was the one that remained — pre-committed stopping criteria and worst-case test footage, which caught the one regression that mattered.

Reusable Lessons

1. Write your stopping criteria before the experiment. "Stop if masked PSNR drops below 36" was decided while neutral. After seeing the +3–4dB victories, drawing that line honestly would have been much harder.
2. Put your worst historical bug in the test set and weaponize it. The gradual drift clip was built to catch chained freezing; it then caught a completely different failure. Worst-case footage appreciates in value over time.
3. Aggregates hide regressions. Subtitles up 3–4dB, drift down 3.6dB — the average says "improvement." The team read per-clip results before looking at means, and the stopping line was per-clip too.
4. Simpler mechanisms fail differently, and not gently. Removing the accumulator eliminated an entire class of proxy-error bugs and introduced none — but it amplified a compensation error the old design had been quietly capping. Budget time for analyzing new failure modes from elegant ideas.
5. When the root cause is small, fix forward. Both fixes were a few lines each, both inside the new design. Rolling back would have meant discarding a real quality gain just to avoid understanding a small bug.

This post is part of the ClearPix engineering series on building free, private, fully browser-native media tools. The anchor skip-frame gate has shipped alongside the video watermark remover and subtitle remover, both running entirely in-browser with no uploads required.