Software testing has changed. The explosion of mobile means the way users consume applications and software is drastically different these days. Lab-based testing doesn’t cover enough of the variables mobile users encounter. People now use mobile devices in places they wouldn’t consume traditional software— the bathroom, social gatherings, shopping, and when commuting .
Developers and testers now have to account for mobile devices with different screen sizes, resolutions, and controls; different operating systems and their versions—sometimes six or more; different network carriers, connections, and connection strengths (which change with location); and myriad apps that might be functioning on any given device, vying for attention and bandwidth. The demands of current software use dictate a change in the way software is tested.
On Dr. Dobbs senior director of product management at Shunra Software Dave Berg outlines one of the biggest shifts:
From city to city, and even sometimes block to block, [mobile] connection and performance is unreliable and constantly shifting. The simple physics of mobile technology results in greater latency variation than a hard-lined connection, making slower, unreliable speeds inherent in a mobile connection.
This was not something you had to account for when testing apps across traditional network conditions. A hard-lined connection, even if it was at a crawling 28.8 Kbps, could be counted on to remain constant. You knew what network conditions your app would encounter, and this made performance testing (relatively) easy.
Assuming a constant level of network performance with mobile is a risky misstep and can leave major issues undetected. Testing apps in a traditional manner without accounting for such mobile network issues as increased and variable latency, jitter, and packet loss means your testing will yield unreliable results and put your app in a position for failure.
“Unreliable results” and incomplete results are the two overarching reasons mobile app testing will be forced outside the traditional setting and into the wild. You simply cannot adequately recreate all the different network connections users will experience. Even if you could, you’d have to replicate that connection on literally hundreds (if not thousands) of devices and dozens of OS versions if you’re looking to release a far-reaching app. The requirements for testing mobile apps are staggering.
The bottom line is testing entirely in a lab just doesn’t cut it anymore. Don’t abandon in-house testing; it can save a lot of time and money if obvious bugs are found in house. But if you expect to release a well-performing, user-friendly, highly rated app, you’re going to have to move a portion of the testing into the real world.
Software testing was never simple. But the way we use software these days—taking it everywhere with us on devices over which developers have no control—is forcing a drastic change. In-the-wild testing is just another technique you can use to address today’s particular and peculiar challenges of testing.
