24 X 7 Automated Stress-injection based System Effectiveness Testing (ASSET)

The complexity of the stress injection process is a critical issue. As in general purpose system testing, the quality of the results is a function of the capability of the stress testing environment to inject stresses in a directed, time-bound and repeatable manner. Providing support for automation of this process ensures that the stress injection occurs in an accurate manner even in real-time systems, such as those found in the electric power domain, where the potential for human error in non-automated systems is great. The automation also ensures that the available resources are exercised at maximum capacity to ensure that as many stresses as possible are injected per unit of time thereby delivering 24/7 stress injection based testing.

S/TDC's ASSET methodology and toolset merges these two concepts and has been successfully used by S/TDC to measure the impact of system level vulnerabilities on the ability of complex, distributed systems to perform their missions. The fundamental techniques of the ASSET methodology have provided quantitative measures of system properties in many critical systems such as air traffic control and railway systems, as well as multiple government research programs.

There are four phases in the ASSET methodology: mission requirements and input gathering to generate measures of effectiveness and utility, stressor selection and test generation, test execution and data collection, analysis and refinement.

Mission, Measures of Effectiveness and Utility
The ASSET process begins with the characterization of the system's mission into system effectiveness requirements. This is accomplished by the systematic decomposition of the system architecture and its corresponding requirements into quantitative Measures of Effectiveness (MOEs) and Measures of Performance (MOPs). This characterization process is closely coupled with the concept of operations (CONOPS) of the system.

Stressor Selection and Test Generation
Armed with a clear understanding of the system architecture and requirements, a Failure Modes Effects Analysis (FMEA) is performed on the system to determine the various system failures that can occur, their potential causes and their corresponding impact on the system's ability to achieve its system effectiveness goal. A critical product of this analysis is the identification of the various faults and fault effects, which in turn, provide the requirements for the set of stressors that need to be used in the stress-injection based tests. The selected set of stressors provides the capability to inject the stress effects of faults that are expected to significantly impact system effectiveness.

The FMEA also provides the information needed to devise effective diagnostic and functional tests of the system to help validate the system effectiveness requirements. Ideally, the tests stress the system with the various faults and errors that are expected to trigger the error detection and recovery mechanisms in the system, thereby validating the effectiveness of that specific mechanism. ASSET tools provide a convenient, reusable means to represent test definitions in a high level scripting language (Ruby), that provides all the advantages, such as rapid prototyping, ready reuse etc., that are attributed to agile scripting languages.

Evidence Collection
Following typical testing paradigms, the functional tests defined in the previous phase are executed in the experimental environment. Typically, multiple tests are scheduled and executed in an automated manner, with the system being reset to nominal values in between tests. Each test injects single or multiple stresses into the system, simultaneously or in sequence as desired by the test objectives. Stress-injection may also be triggered by functional events or the activation of recovery/defense mechanisms in the system. Tests may be run as many times as necessary to ensure that adequate evidence has been obtained. In systems where ASSET is used prior to deployment, the vulnerabilities uncovered by diagnostics tests serve as feedback to the developers and integrators of the system.

The ASSET tools provide an automated environment for the execution of test scripts and collection of test data. The environment has a toolbox of stressors, sensors, scripting libraries and reporting tools that allow for the flexible, controlled and automated introduction of measurement probes and control actuators into the test environment. The resultant stress effects and measured quantitative data fuels the next phase of the ASSET methodology.

Analysis and Refinement
As the number of key metrics identified to evaluate the system expands, a mathematically sound procedure for assessing preferences over the multiple measures becomes essential. This final phase of the ASSET methodology accounts for the multiple dimensions of metrics in large distributed systems using Multi-Attribute Utility (MAU) analysis - a method for performing evaluations with multiple objectives, criteria, or impacts.

In conjunction with domain knowledge experts, the relative importance of the various metrics measured in the system and their impact on system effectiveness is identified. Swing weights are assigned to each metric to reflect the specific system assertions made by the system architects and component developers in the context of mission level goals. The MAU analysis subsequently merges the relative contributions of the various measures to arrive at a quantitative measure representing the overall system effectiveness. As implemented in the DARPA Ultralog program, the ASSET methodology also incorporated the results of an architecture analysis in validating the system survivability claim. The results of this phase of the ASSET methodology guide the system architects and designers in improving the system architecture and design by exposing the vulnerabilities inherent in specific design choices. Furthermore, these results provide quantitative evidence of the merits of alternative mitigation strategies.


For more information about ASSET, please contact us here.