Knight Capital Group is a corporation that uses complex, automated trading systems to provide order execution for 19,000 equities in addition to trading services for options, currencies, and bonds.
On August 1, 2012, Knight Capital suffered a disastrous electronic trading glitch. This bug was introduced the night before the incident during a software upgrade, involving the installation of the NYSE's new Retail Liquidity Program.
Knight CEO Tom Joyce stated that the glitch was a large—although simple—breakdown caused by an issue with trading technology. Knight is doing an internal investigation to identify what exactly happened and to identify how they can enhance the system to prevent this type of incident recurring.
The trading glitch resulted in a catastrophic $440 million dollar loss and prompted inquiries from regulators including the Securities and Exchange Commission. Publicly traded firms are required to adhere to federal regulatory requirements as described in the ISACA Cobit framework including managing changes.
Upgrading a trading environment can be very difficult and potentially cause an outage resulting in downtime for the entire trading system. Every minute that a trading system is unavailable is costly because the market moves quickly, resulting in missed opportunities to trade at a favorable price. The Knight Capital incident was even more serious because it resulted in undesirable trades, highlighting the fact that the system lacked a switch to halt the flood of rapid trades. This resulted in a catastrophic—possibly fatal—loss for the firm.
DevOps is emerging as a set of principles and functions that can help streamline the entire build, package, and deployment of software.
DevOps prevents mistakes by helping development and operations work more closely together, identifying and eliminating risk early in the software and systems lifecycle. Many organizations learn that automating the entire build, package, and deployment process can indeed reduce risk and result in more reliable production deployments.
Better communication between development and operations results in systems that are built and deployed with the needs of runtime operations in mind, which might have helped operations react more swiftly in this incident.
It may take some time to sort out exactly what occurred in the Knight Capital incident, but what is clear is that complex software upgrades need to be performed in accordance with all regulatory guidelines and must make use of industry best practices—including DevOps!
Bob Aiello is a consultant, a technical editor for CM Crossroads, and the author of Configuration Management Best Practices: Practical Methods that Work in the Real World. Bob has served as the vice chair of the IEEE 828 Standards working group and is a member of the IEEE Software and Systems Engineering Standards Committee (S2ESC) management board.