Who's Who in SBSE

Previous research using genetic algorithms to automate the generation of data for path testing has utilized several different fitness functions, assessing their usefulness by comparing them to random generation. This paper describes two sets of experiments that assess the performance of several fitness functions, relative to one another and to random generation. The results demonstrate that some fitness functions provide better results than others, generating fewer test cases to exercise a given program path. In these studies, the branch predicate and inverse path probability approaches were the best performers, suggesting that a two-step process combining these two methods may be the most efficient and effective approach to path testing.

This paper describes two laboratory experiments designed to evaluate a failure-pursuit strategy for system level testing. In the first experiment, two GAs are used to automatically generate test suites that are rich in failure-causing test cases. Their performance is compared to random generation. The resulting test suites are then used to train a series of decision trees, producing rules for classifying other test cases. Finally, the performance of the classification rules is evaluated empirically. The results indicate that the combination of GA-based test case generation and decision tree induction can produce rules with high-predictive accuracy that can assist human testers in diagnosing the cause of system failures.

The potential cost savings from handling software errors within a development cycle, rather than the subsequent cycles, has been estimated at nearly 40 billion dollars by the National Institute of Standards and Technology. This figure emphasizes that current testing methods are often inadequate, and that helping reduce software bugs and errors is an important area of research with a substantial payoff. This is particularly true for the increasingly complex, distributed systems used in many applications from embedded control systems to military command and control systems. These systems may exhibit intermittent or transient errors after prolonged execution that are very difficult to diagnose. This paper explores strategies that combine automated test suite generation techniques with high volume or long sequence testing. Long sequence testing repeats test cases many times, simulating extended execution intervals. These testing techniques have been found useful for uncovering errors resulting from component coordination problems, as well as system resource consumption (e.g. memory leaks) or corruption. Coupling automated test suite generation with long sequence testing could make this approach more scalable and effective in the field.

Highly complex and interconnected systems may suffer from intermittent or transient failures that are particularly difficult to diagnose. This research focuses on the use of genetic algorithms for automatically generating large volumes of software test cases. In particular, the paper explores two fundamental strategies for improving the performance of genetic algorithm test case breeding for high volume testing. The first strategy seeks to avoid evaluating test cases against the real target system by using oracles or models. The second strategy involves improving the more costly components of genetic algorithms, such as fitness function calculations. Together, the various approaches offer opportunities for performance improvements that make these techniques more scalable for realistic applications.

The potential cost savings from handling software errors within a development cycle, rather than subsequent cycles, has been estimated at 38.3 billion dollars. Such figures emphasize that current testing methods are inadequate, and that helping reduce software bugs and errors is an important area of research with a substantial payoff. This paper reports on research using genetic algorithms for test case generation for systems level testing, building on past work at the unit testing level. The goals of the paper are to explore the use of genetic algorithms for testing complex distributed systems, as well as to develop a framework or vocabulary of important environmental attributes that characterize complex systems failures. In addition, preliminary visualization techniques that might help software developers to understand and uncover complex systems failures are explored.

Faulty software is usually costly and possibly life threatening as software becomes an increasingly critical component in a wide variety of systems. Thorough software testing by both developers and dedicated quality assurance staff is one way to uncover flaws. Automated test generation techniques can be used to augment the process, free of the cognitive biases that have been found in human testers. This paper focuses on breeding software test cases using genetic algorithms as part of a software testing cycle. An evolving fitness function that relies on a fossil record of organisms results in interesting search behaviours, based on the concepts of novelty, proximity, and severity. A case study that uses a simple, but widely studied program is used to illustrate the approach. Several visualization techniques are also introduced to analyze particular fossil records, as well as the overall search process.