The lack of a robust software development process can result in outages and impact large-scale financial systems. This problem was evident in the initial public offering (IPO) of Facebook, which began shortly after 11:00 AM on Friday, May 18, 2012.
Unfortunately, the NASDAQ-based IPO of Facebook resulted in trading glitches that caused some investors to unfriend Facebook in a very real and practical way. Reports indicate that part of the problem was NASDAQ's inability to successfully manage the switchover to a backup copy of the order matching engine, resulting in a race condition that impacted the trading platform. As a result, trading execution problems adversely impacted investors by preventing their being able to sell their stock at their planned price.
When financial services firms fail to maintain proper IT controls, they may be held accountable and forced to compensate their stakeholders or others who are impacted by the event.
NASDAQ had to set aside $40 million to compensate investors for trading glitches and failing to provide a fluid and orderly market which is expected in trading environments. One of the people to complain was none other than Knight Capital's CEO Tom Joyce, whose company experienced its own trading problems on August 1, 2012. When the NASDAQ Facebook glitch occurred, CEO Joyce indicated that his company lost $42 million as a direct result of the software and systems issues.
Aside from angering investors, many of whom suffered financial losses, NASDAQ OMX must explain its internal controls to the Securities and Exchange Commission. Federal regulatory requirements oblige many financial services firms, including NASDAQ, to establish and maintain IT controls, including adequate testing along with change and configuration management.
Software outages and glitches that impact customers usually result in increased scrutiny to ascertain if proper controls are in place. DevOps provides the structure to rapidly build, package, and deploy applications. For example, with proper source code management practices, variants in the codebase may be easily created to help facilitate application load testing. The bottom line is that companies need to invest more money and effort into building software and systems that can reliably and effectively handle the peak loads.
Software and systems development best practices can help reduce risk by improving software quality and productivity. This includes having the ability to reliably build, package, and deploy applications. Unfortunately, not all companies understand the importance of establishing robust application development practices including capacity and load testing.
Bob Aiello is a consultant, a technical editor for CM Crossroads, and the author of Configuration Management Best Practices: Practical Methods that Work in the Real World. Bob has served as the vice chair of the IEEE 828 Standards working group and is a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) management board.