Mutation Analysis for Reactive System Environment Properties

2009

2017

Model based testing is a well-established approach to test reactive systems described by formal models. In this paper we look at the problem of generating complete test suite for reactive systems formally described as Input Output Labeled Transition Systems (IOLTSs). In this work we propose a new notion of conformance relation for IOLTS models based on formal languages and automata. We show that this new conformance relation is more general than the well-studied ioco conformance relation. We then describe how to generate test suites that are sound and exhaustive for a given specification model, according to the new conformance relation. We impose no restrictions on the structure of the specification model, a distinct advantage when compared to other recent works.

Computer Languages, Systems & Structures, 2015

Most of today's embedded systems are very complex. These systems, controlled by computer programs, continuously interact with their physical environments through network of sensory input and output devices. Consequently, the operations of such embedded systems are highly reactive and concurrent. Since embedded systems are deployed in many safety-critical applications, where failures can lead to catastrophic events, an approach that combines mathematical logic and formal verification is employed in order to ensure correct behavior of the control algorithm. This paper presents What You Prove Is What You Execute (WYPIWYE) compilation strategy for a Globally Asynchronous Locally Synchronous (GALS) programming language called Safey-Critical SystemJ. SC-SystemJ is a safety-critical subset of the SystemJ language. A formal big-step transition semantics of SC-SystemJ is developed for compiling SC-SystemJ programs into propositional Linear Temporal Logic formulas. These LTL formulas are then converted into a network of Mealy automata using a novel and efficient compilation algorithm. The resultant Mealy automata have a straightforward syntactic translation into Promela code. The resultant Promela models can be used for verifying correctness properties via the SPIN model-checker. Finally there is a single translation procedure to compile both: Promela and C/Java code for execution, which satisfies the De-Bruijn index, i.e. this final translation step is simple enough that is can be manually verified.

Tenth ACM/IEEE International Conference on Formal Methods and Models for Codesign (MEMCODE2012), 2012

Recently, software practitioners, using modelbased engineering and similar methods, have begun developing software from models. After creating a model of the required system behavior, a developer can obtain assurance of the model by validating that it captures the intended behavior and verifying that it satisfies critical properties. Invariants are important to both validation, as a check that the model's behavior matches the intended behavior, and verification, as auxiliaries in proving critical system properties, either automatically or with human guidance. A common approach to discovering invariants is to propose and then check candidate invariants. In contrast, our invariant generation techniques deduce invariants directly from the specification of a model. This paper presents more powerful versions of our earlier techniques for invariant generation and illustrates their utility for a real-world AirLock system.

Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279), 1998

This paper addresses the problem of automatizing the production of test sequences for reactive systems. We particularly focus on two points: (1) generating relevant inputs, with respect to some knowledge about the environment in which the system is intended to run; (2) checking the correctness of the test results, according to the expected behavior of the system. We propose to use synchronous observers to express both the relevance and the correctness of the test sequences. In particular, the relevance observer is used to randomly choose inputs satisfying temporal assumptions about the environment. These assumptions may involve both Boolean and linear numerical constraints. A prototype tool, called Lurette, has been developed and experimented, which works on observers written in the Lustre programming language.

International Journal on Software Tools for Technology Transfer, 2014

The goal of the RERS challenge is to evaluate the effectiveness of various verification and validation approaches on reactive systems, a class of systems that is highly relevant for industrial critical applications. The RERS challenge brings together researchers from different areas of software verification and validation, including static analysis, model checking, theorem proving, symbolic execution, and testing. The challenge provides a forum for experimental comparison of different techniques on specifically designed verification tasks. These benchmarks are automatically synthesized to exhibit chosen properties, and then enhanced to include dedicated dimensions of difficulty, such as conceptual complexity of the properties (e.g., reachability, safety, liveness), size of the reactive systems (a few hundred lines to millions of lines), and complexity of language features (arrays and pointer arithmetic). The STTT special section on RERS describes the results of the evaluations and the different analysis techniques that were used in the RERS challenges 2012 and 2013.

2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, 2011

Conformance testing amounts to verifying adequacy between the behaviors and the specified behaviors of an implementation. In this paper, we handle model-based conformance testing for data-flow critical systems with time constraints. Specifications are described with a formal model adapted for such systems and called Variable Driven Timed Automata (VDTA). VDTA are inspired by timed automata but they use input/output communication variables, allowing clear and short specifications. We present a conformance relation for this model and we propose a symbolic test selection algorithm based on a test purpose. The selection algorithm computes the variations on inputs allowing to reach an expected state of the implementation. Then we propose an on-line testing algorithm.

2001

Testing commercial software is expensive and time consuming. Automated testing methods promise to save a great deal of time and money throughout the software industry. One approach that is well-suited for the reactive systems found in telephone switching systems is speci cation-based testing. We have built a set of tools to automatically test software applications for violations of safety properties expressed in temporal logic. Our testing system automatically constructs nite state machine oracles corresponding to safety properties, builds test harnesses, and integrates them with the application. The test harness then generates inputs automatically to test the application. We describe a study examining the feasibility of this approach for testing industrial applications. To conduct this study we formally modeled an Automatic Protection Switching system (APS), which is an application common to many telephony systems. We then asked a number of computer science graduate students to develop several versions of the APS and use our tools to test them. We found that the tools are very e ective, save signi cant amounts of human e ort (at the expense of machine resources), and are easy to use. We also dis

Science of Computer Programming, 2016

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission, are exemplarily applied to a hierarchy of graph-based models of increasing expressive power including directed graphs, event sequence graphs, finite-state machines and statecharts. Test cases generated based on the mutated models (mutants) are used to determine not only whether each mutant can be killed but also whether there are any faults in the corresponding system under consideration (SUC) developed based on the original model. Novelties of our approach are: (1) evaluation of the fault detection capability (in terms of revealing faults in the SUC) of test sets generated based on the mutated models, and (2) superseding of the great variety of existing mutation operators by iterations and combinations of the two proposed elementary operators. Three case studies were conducted on industrial and commercial real-life systems to demonstrate the feasibility of using the proposed MBMT approach in detecting faults in SUC, and to analyze its characteristic features. Our experimental data suggest that test sets generated based on the mutated models created by insertion operators are more effective in revealing faults in SUC than those generated by omission operators. Worth noting is that test sets following the MBMT approach were able to detect faults in the systems that were tested by manufacturers and independent testing organizations before they were released.

2023

Developing reliable reactive software is notoriously di cult -particularly when that software reacts by changing its behavior. Some of this di culty is inherent; software that must respond to external events as they arrive tends to end up in states that are dependent on the value of that input and its order of arrival. This results in complicated corner cases that can be challenging to recognize. However, we nd that some of the complexity is an accident of the features of the programming languages widely used in industry. The loops and subroutines of structured programming are well-suited to data transformation, but poorly capture -and sometimes obscure -the ow of data through reactive programs developed using the inversion-of-control paradigm; an event handler that modi es the data ow tends to be declared closer to the de nition of the event that activates it than to the initial denition of the data ow that it modi es. This paper approaches both challenges with a language inspired by the declarative modules of languages SIGNAL and Lustre and the semantics of the SodiumFRP Functional Reactive Programming library with a declarative mechanism for self modi cation through module substitution. These language features lead to software with a code structure that closely matches the ow of data through the running program and thus makes software easier to understand. Further, we demonstrate how those language features enable a bounded model checking approach that can verify that a reactor meets its requirements or present a counterexample trace, a series of states and inputs that lead to a violation. We analyze the runtime performance of the veri er as a function of model size and trace length. CCS Concepts: • Software and its engineering → Data ow languages.