Lutess

Journal of Systems and Software, 1998

Testing commercial software is expensive and time consuming. Automated testing methods promise to save a great deal of time and money throughout the software industry. One approach that is well-suited for the reactive systems found in telephone switching systems is speci cation-based testing. We have built a set of tools to automatically test software applications for violations of safety properties expressed in temporal logic. Our testing system automatically constructs nite state machine oracles corresponding to safety properties, builds test harnesses, and integrates them with the application. The test harness then generates inputs automatically to test the application. We describe a study examining the feasibility of this approach for testing industrial applications. To conduct this study we formally modeled an Automatic Protection Switching system (APS), which is an application common to many telephony systems. We then asked a number of computer science graduate students to develop several versions of the APS and use our tools to test them. We found that the tools are very e ective, save signi cant amounts of human e ort (at the expense of machine resources), and are easy to use. We also dis

Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257), 1998

Lutess is a tool that we developed for testing reactive synchronous software and which is being used in different industrial contexts. It offers several formal testing methods with automatic generation of test data from the environment specification. Lutess provides also an approach to assign a probability to the next event issued by the environment. However, a Lutess' user faces the problem of expressing operational profiles with sophisticated and varying probability assignments to software inputs, in a format directly usable by Lutess. The paper concentrates on how to express operational profiles for specifications which are implemented as Binary Decision Diagrams.

2007

Lutess is a testing environment designed for synchronous software specified with Lustre, a synchronous data-flow language widely used in safety critical domains such as avionics, energy and transport. Lutess automatically transforms the formal description of the program environment and properties to test generators that feed, on the fly, the program under test. A new version of Lutess has been recently developed, using Constraint Logic Programming. In this version, it is possible to take into account numeric input and output variables and to introduce hypotheses on the program under test. The input language of Lutess has been consequently extended. In this paper we present the new set of operators of the language and illustrate their execution semantics on a simple example.

G∀ST is a fully automatic test system. Given a logical property, stated as a function, it is able to generate appropriate test values, to execute tests with these values, and to evaluate the results of these tests. Many reactive systems, like automata and protocols, however, are specified by a model rather than in logic. There exist tools that are able to test software described by such a modelbased specification, but these tools have limited capabilities to generate test data involving data types. Moreover, in these tools it is hard or even impossible to state properties of these values in logic. In this paper we introduce some extensions of G∀ST to combine the best of logic and model based testing. The integration of model based testing and testing based on logical properties in a single automated system is the contribution of this paper. The system consists only of a small library rather than a huge stand-alone system.

Proceedings 19th IEEE Real-Time Systems Symposium (Cat. No.98CB36279), 1998

This paper addresses the problem of automatizing the production of test sequences for reactive systems. We particularly focus on two points: (1) generating relevant inputs, with respect to some knowledge about the environment in which the system is intended to run; (2) checking the correctness of the test results, according to the expected behavior of the system. We propose to use synchronous observers to express both the relevance and the correctness of the test sequences. In particular, the relevance observer is used to randomly choose inputs satisfying temporal assumptions about the environment. These assumptions may involve both Boolean and linear numerical constraints. A prototype tool, called Lurette, has been developed and experimented, which works on observers written in the Lustre programming language.

2020

Reactive systems are characterized by the interaction with the environment, where the exchange of the input and output stimuli, usually, occurs asynchronously. Systems of this nature, in general, require a rigorous testing activity over their developing process. Therefore model-based testing has been successfully applied over asynchronous reactive systems using Input Output Labeled Transition Systems (IOLTSs) as the basis. In this work we present a reactive testing tool to check conformance, generate test suites and run test cases using IOLTS models. Our tool can check whether the behavior of an implementation under test (IUT) complies with the behavior of its respective specification. We have implemented a classical conformance relation \ioco and a more general notion of conformance based on regular languages. Further, the tool provides a test suite generation in a black-box testing setting for finding faults over IUTs according to a specific domain. We have also described some cas...

Three approaches to the problem of testing synchronous data-flow programs written in LUSTRE are presented. LUSTRE is a language well-adapted to both the specification and the development of reactive software. The first approach automatically transforms a set of LUSTRE invariant properties characterizing the environment of the reactive program into a constrained random generator of test data sequences. The second approach consists in analyzing the required safety properties (written in LUSTRE) of the software. This analysis may result, in specific cases, in automatic generators of relevant test data. The third approach considers that LUSTRE is used for the implementation of the software, and defines structure-based testing criteria. These criteria are specific to the model which captures the LUSTRE program structure : the operator net. Moreover, an automatic test data generation process is described for this last approach using LESAR, a tool designed to automatically prove the satisfaction of safety properties on LUSTRE programs.

Lecture Notes in Computer Science, 2006

Although computer systems penetrate all facets of society, the software running those systems may contain many errors. Producing high quality software appears to be difficult and very expensive. Even determining the quality of software is not easy. Testing is by far the most used way to estimate the quality of software. Testing itself is not easy and time consuming. In order to reduce the costs and increase the quality and speed of testing, testing should be automated itself. An automatical specification based test tool generates test data, executes the associated tests, and makes a fully automatically verdict based on a formal specification. Advantages of this approach are that one specifies properties instead of instances of these properties, test data are derived automatically instead of manually, the tests performed are always up to date with the current specification, and automatic testing is fast and accurate. We will show that functions in a functional programming language can be used very well to model properties. One branch of the automatic test system Gast handles logical properties relating function arguments and results of a single function call. The other branch of Gast handles specifications of systems with an internal state.

2013 8th IEEE International Symposium on Industrial Embedded Systems (SIES), 2013

Automating the functional testing of reactive systems requires to provide a formal specification of the system environment which defines the admissible test inputs. It also requires a specification of the expected properties of the system in order to decide whether a test succeeds or fails. Engineering these formal specifications is a difficult task, as it is not part of the usual manual testing process. In this paper, we report some experiments that have been conducted within a project in collaboration with industrial developers of nuclear power plant control systems. In this project, automatic testing tools have been used for checking the correctness of reactive systems developed incrementally, using heterogeneous industrial engineering workbenches. But these tools appeared to be useful also for elaborating and refining formal, consistent, and accurate functional requirements.

Theoretical Computer Science, 2005

Software testing is typically an ad-hoc process where human testers manually write test inputs and descriptions of expected test results, perhaps automating their execution in a regression suite. This process is cumbersome and costly. This paper reports results on a framework to further automate this process. The framework consists of combining automated test case generation based on systematically exploring the input domain of the program with runtime verification, where execution traces are monitored and verified against properties expressed in temporal logic. The input domain of the program is explored using a model checker extended with symbolic execution. Properties are formulated in an expressive temporal logic. A methodology is advocated that generates properties specific to each input instance rather than formulating properties uniformly true for all inputs. Capabilities for analyzis of concurrency errors are planned to be integrated with temporal logic monitoring. The paper describes an application of the technology to a planetary rover controller.