The state problem for test generation in Simulink

Proceedings of the 7th ACM international conference on Embedded software - EMSOFT '08, 2008

The Simulink/Stateflow (SL/SF) environment from Mathworks is becoming the de facto standard in industry for model based development of embedded control systems. Many commercial tools are available in the market for test case generation from SL/SF designs; however, we have observed that these tools do not achieve satisfactory coverage in cases when designs involve nonlinear blocks and Stateflow blocks occur deep inside the Simulink blocks. The recent past has seen the emergence of several novel techniques for testing large C, C++ and Java programs; prominent among them are directed automated random testing (DART), hybrid concolic testing and feedback-directed random testing. We believe that some of these techniques could be lifted to testing of SL/SF based designs; REDIRECT (RandomizEd DIRECted Testing), the proposed testing method of this paper, is an attempt towards this direction. Specifically, REDIRECT uses a careful combination of the above techniques, and in addition, the method uses a set of pattern-guided heuristics for tackling nonlinear blocks. A prototype tool has been developed and the tool has been applied to many industrial strength case studies. Our experiments indicate that a careful choice of heuristics and certain combinations of random and directed testing achieve better coverages as compared to the existing commercial tools. 1

2015

Matlab, Simulink/Stateflow Model is the most extensively used industrial tools that include system development that allows models to be developed, visualized and exercised. Matlab, Simulink/Stateflow (SL/SF) is used particularly for developing embedded system. Usually SL/SF models are hierarchical;we can split up any complex system into trivial parts that can be viewed as a level of abstraction that helpful in understanding the system easily. The resulting model must be tested in order to find fault in the system, especially embedded system. In order to doing the model based testing of the SL / SF model, we have primarily developed any SL/SF system, subsequent we have to convert that model into xml file, succeeding using XML parser and our proposed algorithm we have to develop SL/SF System dependency graph (SSDG) for SL/SF model, subsequent using SSDG, we apply our algorithm to generate test sequences and test cases for the SL/SF model.

Journal of Computer Science and Cybernetics, 2017

Mutation testing is a powerful and effective software testing technique to assess the quality of test suites. Although many research works have been done in the field of search-based testing, automatic test data generation based on the mutation analysis method is not straightforward. In this paper, an Improved Genetic Algorithm (IGA) is proposed to increase the quality of test data based on mutation coverage criterion. This algorithm involves some modifications of genetic operators and the employment of memory mechanism to enhance its effectiveness. The proposed approach is implemented to generate test data for Simulink models. The obtained results indicated that IGA outperformed the conventional genetic algorithm in terms of the quality of test sets, and the execution time.

2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017

One promising way to improve the accuracy of fault localization based on statistical debugging is to increase diversity among test cases in the underlying test suite. In many practical situations, adding test cases is not a cost-free option because test oracles are developed manually or running test cases is expensive. Hence, we require to have test suites that are both diverse and small to improve debugging. In this paper, we focus on improving fault localization of Simulink models by generating test cases. We identify three test objectives that aim to increase test suite diversity. We use these objectives in a search-based algorithm to generate diversified but small test suites. To further minimize test suite sizes, we develop a prediction model to stop test generation when adding test cases is unlikely to improve fault localization. We evaluate our approach using three industrial subjects. Our results show (1) the three selected test objectives are able to significantly improve the accuracy of fault localization for small test suite sizes, and (2) our prediction model is able to maintain almost the same fault localization accuracy while reducing the average number of newly generated test cases by more than half.

arXiv (Cornell University), 2022

Simulation-based software testing supports engineers in finding faults in Simulink ® models. It typically relies on search algorithms that iteratively generate test inputs used to exercise models in simulation to detect design errors. While simulation-based software testing techniques are effective in many practical scenarios, they are typically not fully integrated within the Simulink environment and require additional manual effort. Many techniques require engineers to specify requirements using logical languages that are neither intuitive nor fully supported by Simulink, thereby limiting their adoption in industry. This work presents HECATE, a testing approach for Simulink models using Test Sequence and Test Assessment blocks from Simulink ® Test TM. Unlike existing testing techniques, HECATE uses information from Simulink models to guide the search-based exploration. Specifically, HECATE relies on information provided by the Test Sequence and Test Assessment blocks to guide the search procedure. Across a benchmark of 16 Simulink models from different domains and industries, our comparison of HECATE with the state-of-the-art testing tool S-TALIRO indicates that HECATE is both more effective (more failurerevealing test cases) and efficient (less iterations and computational time) than S-TALIRO for ≈94% and ≈81% of benchmark models respectively. Furthermore, HECATE successfully generated a failure-revealing test case for a representative case study from the automotive domain demonstrating its practical usefulness.

Electronic Notes in Theoretical Computer Science, 2007

Model-based testing has been proposed as a technique to automatically verify that a system conforms to its specification. A popular approach is to use a model-checker to produce a set of test cases by formulating the test generation problem as a reachability problem. To guide the selection of test cases, a coverage criterion is often used. A coverage criterion can be seen as a set of items to be covered, called coverage items. We propose an on-the-fly algorithm that generates a test suite that covers all feasible coverage items. The algorithm returns a set of traces that includes a path fulfilling each item, without including redundant paths. The reachability algorithm explores a state only if it might increase the total coverage. The decision is global in the sense that it does not only regard each individual local search branch in isolation, but the total coverage in all branches together. For simpler coverage criteria as location of edge coverage, this implies that each model state is never explored twice. The algorithm presented in this paper has been implemented in the test generation tool UPPAAL CO ER. We present encouraging results from applying the tool to a set of experiments and in an industrial sized case study.

Software Testing, Verification and Reliability, 2013

Model-based test generation techniques based on random input generation and guided simulation do not satisfy the demands of high test coverage and completeness guarantees as required by safety-critical applications. Recently, test generation techniques based on model checking have been reported to bridge this gap. To evaluate the effectiveness of these techniques, an in-house tool suite, AutoMOTGen, has been developed for Simulink/Stateflow and applied on real-life case studies at General Motors. This paper outlines the test generation methodology of AutoMOTGen and gives a comparative study with a commercial, primarily random input-based, test generation tool on the same set of examples. The results indicate that in terms of coverage, model checking-based techniques complement the random input-based techniques. In addition, they provide proofs for unreachability that can aid in debugging the models. Therefore, it is recommended that model checking-based tools be utilized to complement and enhance the effectiveness of model-based testing methods in safety-critical systems engineering.

Annual Software Engineering Workshop, 2005

The Simulink model tester "global assertions" capability developed by the software productivity consortium and T-VEC technologies is described in this paper. This assertions mechanism incorporates design assumptions or natural relationships with Simulink models to automatically generate more valid test vectors using the test automation framework. The positive impact this mechanism has on development and testing of Simulink models for the Lockheed Martin Space System's independent verification and validation team is also discussed

IEEE Transactions on Software Engineering, 2019

Model-based test generation by model checking is a well-known testing technique that, however, suffers from the state explosion problem of model checking and it is, therefore, not always applicable. In this paper, we address this issue by decomposing a system model into suitable subsystem models separately analyzable. Our technique consists in decomposing that portion of a system model that is of interest for a given testing requirement, into a tree of subsystems by exploiting information on model variable dependency. The technique generates tests for the whole system model by merging tests built from those subsystems. We measure and report effectiveness and efficiency of the proposed decomposition-based test generation approach, both in terms of coverage and time.

2021

Mutation testing offers stronger test adequacy than conventional structural coverage metrics. It employs mutants, which can mimic the real software faults, to measure the fault detection capacity of a given test suite. Mutation testing has been shown to work on an industrial scale for embedded systems, however only for unit tests. Since it is common practice to use model-based design to analyze, and test embedded software, it is worthwhile to investigate mutation testing in the context of MATLAB/Simulink. In this paper, we propose a framework for mutation testing of Simulink models. We validate the framework by means of a line follower robot constructed with Lego/Mindstorm and modelled using a series of Simulink models. This study also addresses possible open research and development challenges.