Sometimes, in order to get everyone on board with adopting test automation, you have to build a business case to highlight the potential benefits. Once you’ve convinced the organization to make that investment, you should determine whether it’s actually yielding the predicted benefits—and you’ll want to keep these benefits visible to key stakeholders to reinforce the value.
We tend to have many metrics in place to track the progress of testing and measure the degree of readiness or risks in our software products. These metrics include test effectiveness, software quality, test status, resources, issues, and so forth, but what about metrics for the test automation platform? How do we frame the quantitative and qualitative benefits of test automation in a way that links to the organizational objectives and business goals? This is an important element of planning and implementing our automation architecture.
One of the approaches that has worked well for organizations I’ve worked with is a test automation dashboard. A useful dashboard should reflect values that align with the organization and business goals, be agile and adaptable, and contain information that is actionable and meaningful, with just enough information that it’s still easy to use and maintain.
It’s important to design the dashboard at the beginning of your automation project in order to ensure that the approaches and tools you choose will yield the measures and metrics you need. Five dimensions will usually cover the range of information that the team and key stakeholders require, but of course you can trim or expand these to meet your needs:
Capabilities include the automation platform features, interface capabilities, integrations, types of systems under tests that are supported, and scripting languages supported.
Usage includes the number of users, teams, and projects; the frequency of use of a given automation feature; the rate of automation adoption; and the frequency of use of test suites.
Benefits are metrics such as rate of unit test, story test, and nonfunctional automation; ease of learning and use of the automation platform; scripting or execution efficiency; degree of automation suite code coverage; speed and efficiency of the test automation infrastructure; hours saved versus manual testing; number of additional test cycles achieved per sprint; percent of increased requirements, functional or nonfunctional testing, or structural coverage achieved; and how much earlier defects are found.
Challenges capture the constraints or issues with the tools or automation platform.
Investment encompasses the costs (in time or money), such as defects in the automation platform, licenses, platform maintenance, training and retraining, operation (including hosting, servers, etc.), the time to build, maintain, and execute automated tests, and the time to investigate defects in the automation test infrastructure or test suites.
The dimensions and associated metrics that are applicable can be displayed in a graphical dashboard format, with simple or more complex measures associated with each “dial.” I find trend measures more useful than absolutes.
The critical element of any dashboard is ensuring that it’s aligned with the organization's goals and business objectives, and having just enough measures in place to guide the continual refinement of the automation platform.
