Cross-Platform App QA in Practice: One Developer's Real-Device Testing Strategy

Why Real-Device Testing Can't Be Replaced by Automation

In the world of app development, automated testing is often treated as a silver bullet for quality assurance. But a developer's experience shared on Reddit reveals an underappreciated truth: many platform-specific issues simply can't be caught by automated tests.

This developer maintains a player app called JellyBox, compatible with three major media server backends — Jellyfin, Emby, and Navidrome — and needs it to run reliably across all major platforms. He's candid about the problem: "pretending" you've tested something with automation often means missing real compatibility issues that only surface on actual devices.

This point cuts to the core challenge of cross-platform development. UI rendering differences, system permission behavior, audio/video codec support, network protocol implementations — all of these are highly dependent on specific OS versions and hardware environments. Simulators and virtual environments in CI pipelines simply can't reproduce the edge cases users encounter in the real world.

A Full-Platform Real-Device Test Matrix

To achieve genuine cross-platform validation, this developer assembled a remarkably complete device lineup that covers virtually every environment a consumer user might have.

On the desktop side, he uses an M3 MacBook as his primary Apple Silicon test machine, while keeping an older Intel MacBook — now running Omarchy (an Arch-based Linux configuration) — to verify behavior across different architectures. Windows coverage comes from a physical machine, on which he runs Debian 13, Ubuntu, and Kubuntu via virtual machines, covering mainstream Linux distributions and different desktop environments.

Mobile coverage comes from two phones: an older iPhone 16 Pro Max handles iOS, while his daily driver Pixel 10 Pro XL covers Android. He also occasionally borrows his wife's iPad to test tablet form factor behavior.

The value of this matrix lies in covering three dimensions simultaneously: architecture (ARM/x86), operating system (macOS/Windows/Linux/iOS/Android), and device form factor (desktop/tablet/phone). This combination maximizes the exposure of issues caused by platform differences.

Orthogonal testing is the core methodology behind this device matrix. The basic idea: when there are many test variables, you don't need to exhaustively cover every combination. Instead, you select an "orthogonal" subset where every pair of variable values appears at least once, achieving the broadest coverage of differences with the fewest test cases. In this case, architecture (ARM/x86), operating system (macOS/Windows/Linux/iOS/Android), and form factor (desktop/tablet/phone) form three dimensions whose combinations could produce a huge number of scenarios — but through careful device selection, the developer covers the vast majority of meaningful combinations with roughly six devices. This approach is especially practical for small, resource-constrained teams, helping avoid the trap of "piling up similar devices" while missing the actual points of differentiation.

Validating Multi-Protocol Casting with Real Speakers

For a media player, audio output and casting protocols are unavoidable testing priorities. The developer's setup here is particularly clever.

He uses KEF LSX 2 LT powered speakers paired with an older Harman Kardon subwoofer. The critical reason for choosing this setup isn't just sound quality — it's that the KEF LSX 2 LT simultaneously supports three connection protocols: AirPlay, Chromecast, and DLNA.

This means he can use the same hardware to verify compatibility and stability across different casting protocols — AirPlay for the Apple ecosystem, Chromecast for the Google ecosystem, and DLNA as the more universal open standard. For a media app that needs to work across ecosystems, this "one device, multiple protocols" testing capability dramatically simplifies the validation process.

AirPlay, Chromecast, and DLNA differ significantly in their technical implementations. AirPlay is Apple's proprietary protocol, using RTSP/RTP to stream audio and video, with high network latency sensitivity and a requirement that devices be on the same Apple account or trusted local network. Chromecast (also known as Google Cast) has the sender pass a media URL to the receiver, which then pulls the stream itself — a different traffic path from AirPlay, with compatibility issues more often appearing at the media format negotiation stage. DLNA is based on the older UPnP/SOAP standards; its longer history means vendor implementations vary widely, making device discovery failures and unsupported format errors common edge cases. Supporting all three protocol stacks simultaneously means dealing with three different sets of error patterns, timeout behaviors, and device compatibility matrices — precisely why no single simulator can cover them all.

The QA Philosophy of a Small Studio

Behind this testing setup lies a pragmatic strategy for independent developers and small teams working with limited resources: rather than chasing impressive coverage numbers, focus finite effort on the real-world scenarios most likely to cause problems.

The return on investment is worth considering. Building a device matrix like this carries real costs, but for an app that ships on multiple platforms and depends on strong word-of-mouth to survive, a single serious platform compatibility bug could cause user churn far more expensive than the hardware investment.

The developer himself emphasizes that sharing this approach isn't about promoting any product — it's purely about exchanging ideas on how he does QA for his code. He ends his post with an open question: "How do you test your apps?" — a question every cross-platform developer deserves to think carefully about.

Lessons for Independent Developers

From this case study, several practices emerge with broad value for cross-platform development:

Real devices first, automation as support. Automated testing is well-suited for regression validation and logic checks, but platform-specific rendering, permissions, and protocol issues still need real-device confirmation.

Choose devices for dimensional coverage. Rather than stacking similar devices, apply orthogonal selection across architecture, OS, and form factor dimensions — covering the widest range of differences with the fewest devices.

Make the most of multi-function peripherals. Devices like the KEF LSX 2 LT that support multiple protocols can cover multiple test scenarios in one unit — a smart choice when budgeting carefully.

Most independent developers won't need to replicate this exact configuration, but the core principle — use real environments to expose real problems — is worth borrowing for any team that takes application quality seriously.