Testing Nonfunctional Requirements in an Agile Lifecycle

Effective nonfunctional testing is often the forgotten testing task. It can be the elephant in the room that no one wants to talk about.

To start with, nonfunctional requirements (NFRs)—such as performance, reliability, installability, scalability, usability, security, availability, and so forth—are hard to quantify. Just as importantly, finding someone who has the depth and breadth of knowledge to define NFRs is usually a challenge.

In a traditional waterfall cycle, we may have some definitive NFRs in the system or requirements specification. In safety-critical systems, NFRs are much better defined, but for many commercial software development environments, NFRs remain elusive and tend to be a "late in the lifecycle" testing activity.

As organizations embrace agile, NFRs may be an even greater challenge—both functional and nonfunctional requirements must be considered and validated in each (short) sprint. Ideally, NFRs are a continuous focus during each sprint, instead of being left to the last one.

Here are some ways to better address NFRs in your agile development lifecycle.

Defining nonfunctional requirements:

For brand-new products, use analysis, experiments, and comparative testing, and collect or test user expectations.

• Analyzing the technical, data, and network architecture will yield important characteristics and constraints.
• Design experiments and do comparative testing to evaluate and select the best alternatives.
• Speaking with users and asking successively detailed questions can help quantify NFR expectations. For instance, if customers say, "I need it to go fast," you can further define what they mean by fast. You also can demonstrate the product to users and get their feedback immediately, such as by saying, "Let me load this home page for you, and you tell me if it loaded quickly enough."

For evolving products, use analytics tools to mine existing application and system data.

• Analyze data from customer usage—clicks, workflows, feature usage, errors, abandons, and so forth.
• Analyze information from production systems, including resource usage (memory, process, CPU, and input/output rates), network traffic characterization, security attack attempts, and database statistics.
• Review help desk and issue tickets for outages, defects, enhancement requests, and user errors.
• Use video and screen-capturing software to record user sessions and analyze problem areas.

Documenting nonfunctional requirements:

In waterfall methodologies, NFRs are included in a separate document or category in the repository. But in agile, NFRs normally cut across several user stories, or even the entire system, sp require some special consideration.

• First, ensure that NFRs are being discussed and defined during the agile release or theme planning and that a conversation around how to test these characteristics is started.
• Define scenarios that cut across stories. A scenario describes a real-world example of how people or organizations interact with a system. Scenarios then act as constraints across stories, and they have rules and restrictions that must be met.
• Alternatively, create backlog items for each NFR and link user stories to these NFR backlog items. These will later be estimated and scheduled into the appropriate sprint.

Continuously testing both functional and nonfunctional characteristics across sprints is the key to delivering a minimum viable product.