Just Enough Testing at Each Stage of Your Delivery Pipeline
The continuous delivery pipeline is the process of taking new or changed features from developers and getting them deployed into production and delivered quickly to the customer. To reach that goal, the pipeline must be designed to determine whether the software is a viable candidate for production—and if not, then why.
Having frequent quality gates that give feedback about the quality of the software along the way helps us find that answer quicker. Quality gates are decision points along the pipeline where we gather feedback and decide if we want to keep testing. If the software doesn’t pass a quality gate, then it isn’t worth continuing with that build, and we go back to the developers (or to the requirements) for a fix.
Just like in continuous integration, we want the quickest feedback loops to be earliest in the pipeline. Tests later in the pipeline can be more expensive in terms of time, resources, or effort because they will focus on software we have more confidence in. Although it is tempting to arrange the delivery pipeline in phases (e.g., functional tests, then acceptance tests, then load and performance tests, then security tests), this can lead to problems progressing far down the pipeline before they are caught because we hadn’t done any of the type of testing that would have found them.
So instead, we want to arrange our quality gates so each round provides just enough testing at each stage to give you confidence that the next set of tests is worth the investment. This fits with the theme of “small increments” that recurs in agile methodologies, continuous integration, and continuous delivery.
If unit tests don’t pass, we stop the pipeline because we know the code is broken. Running functional tests would be wasted effort. So unit tests are an obvious quality gate to move early in the pipeline because we get fast feedback.
At the other end of the pipeline, we don’t want to run a full load and performance test before we have confidence that it is worth it. So maybe we could watch the elapsed time for a small handful of functional tests. If those tests take significantly longer than expected, then we know performance has degraded without our investing in a full performance test.
Later in the pipeline (perhaps after a little secure testing, for instance) we can follow up with some more performance tests, all leading to the end stage of the pipeline, when we are confident that it is worth investing in a full load and performance test.
Implementing quality gates encourages agile feedback, which can minimize risk, build quality directly into the project, and, ultimately, reduce development time—by getting it done right the first time.
Gene Gotimer is presenting the session Testing in a Continuous Delivery Pipeline: Faster, Better, Cheaper at STARWEST 2016, October 2–7 in Anaheim, California.
