Stop Releasing Untested Defect Fixes into Production

Releasing untested defect fixes into production is a very real possibility. Being aware of how this can occur may help reduce the possibility of it happening.

Let’s explore the common reasons defect fixes go untested, as well as the steps we can take to prevent them.

Insufficient Testing

Insufficient testing can be characterized by performing the bare minimum of testing and only using the originally written defect description as your guide for the fix verification—in essence, only confirming one specific path through the software.

Sometimes it’s necessary to take a holistic approach to defect fix verification by not just verifying the defect fix as it was written, but all the possible interrelationships of the defect fix to the rest of the software. This may include the need to create additional test data and scenarios and to test interrelated functionalities, various platforms, environmental settings, and nonfunctional tests.

Inadequate Resources

Inadequate test resources include processes, people, tools, and time constraints that constrict and limit the depth of your defect fix verification. For example:

• A lack of testing processes could lead to an ad hoc fashion of defect writing and testing that is directionless, with no guarantee of fully understanding the expected software behavior
• Neglecting the defect tracking database could result in fixed defects going untested, either by never verifying a fixed defect or inadvertently closing a fixed defect before it was tested
• Testers who are insufficiently trained or don’t possess the aptitude for testing may lack needed technical skills
• Environments and platforms that don’t fully represent the customer base restrict the extensiveness of the testing
• Prioritizing test automation and scripting every scenario doesn’t leave enough time for thorough exploratory testing
• Time constraints compress the test effort, so you end up only confirming the defect fix works but don’t have time for any type of regression tests

Undocumented Fixes

A defect fix typically originates from one of three sources: a formally written defect report, some form of change notice or updated requirement, or the developer uncovering an issue during code development or analysis and making a change. This last type is an undocumented update the test team may never know about or get to verify.

Steps for Rectifying

Eliminating the possibility of releasing untested defect fixes into production is doable with a few practical guidelines.

• Realizing that process is necessary: Applying process plasticity, which means the process should be malleable, suitably adjusted for the team and product under test
• Ensuring a competent test team: Employing individuals dedicated to testing
• Defining the word “test”: Identifying within the defect report how the fix was verified, including if any regression, performance, scaling, usability, or security testing that was performed, along with the environments, parameter settings, and any special customer specific builds—as well as what wasn’t tested
• Eliminating undocumented fixes: Any defect uncovered by a developer should be entered in the system, following the same standards as if a tester discovered the issue

Being aware of how untested defect fixes get released into production is the first step to curtailing the problem.