What Is Continuous Testing All About?
Is seems like the term continuous testing burst onto the scene recently, but it’s been bubbling around our industry for at least five years. It started when DevOps got hot as organizations began trying to figure out how to make everything in the software delivery process more continuous and testers felt they were being left out of the DevOps movement.
If you want to get started with continuous testing and need a primer, here are three things you should know.
There’s No Testing Phase
The first goal of continuous testing is to continuously test code as it is created. With continuous deployment, where every code change goes straight into production, we need testing to be fully integrated into the development process so that each code change is tested within its unit, in combination with other units and components, as well as regression tested to make sure it didn’t break something else.
To make this work, software development and testing need to be done together in real time. Whether this means testing is now the sole responsibility of software developers, there are dedicated software developers in test (SDETs) working with each developer, or developer-tester pairs collectively build and test every day, testing must be done completely differently from in the past.
Shift Left to Shorten Feedback Loops
Agile and DevOps are all about shortening feedback loops to make sure our teams are heading in the right direction and to catch errors as early as possible. Shifting left focuses on moving software testing activities as close to code implementation as possible to help in this defect detection and correction. In addition, it is driving the movement to get nonfunctional testing activities integrated much earlier in the software lifecycle so they are not hurdles for delivery either.
Much like having no testing phase may shift the responsibility of some functional testing activities, testing nonfunctional requirements may change too. For instance, performance analysis of the code should be a quality gate that software developers are responsible for during continuous integration of new code. No longer do we wait until the end of the delivery process to find out our code is too slow.
Shift Right to Test Using Real Customer Data
As much as we hear about shifting left, there is a growing buzz around shifting testing right, too. Shifting some testing activities into production provides an opportunity for this testing to be done on real customer data and production platforms that is sometimes difficult to do earlier. It also can help automatically detect and eradicate production issues quickly by shortening the turnaround time on production defects.
Of course, we want to be careful about shifting too much of our testing process to the right for mission-critical applications, as the cost of some defects is so high that we cannot afford for our customers to find them. A safety-critical bug in a pacemaker is an example. We can’t afford to find these bugs in production, so extensive testing must be performed prior to delivery.
