Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Conclusion
References
All Topics
Computer Science

A Lightweight Technique to Identify Equivalent Mutants

Profile image of Beatriz SouzaBeatriz Souza

2020

visibility

…

description

10 pages

link

1 file

Abstract

Mutation analysis is a popular but costly approach to assess the quality of test suites. Equivalent mutants are useless and contribute to increase costs. We propose a lightweight technique to identify equivalent mutants by proving equivalences with Z3 in the context of weak mutation testing. To evaluate our approach, we apply our technique for 40 mutation targets (mutations of an expression or statement) and automatically identify 13 equivalent mutations for seven mutation targets. We manually confirm that the equivalent mutants detected by our technique are indeed equivalent. Moreover, we evaluate our approach in the context of strong mutation testing against mutants generated by MUJAVA for 5 projects. Our technique detects all equivalent mutants detected by TCE. The results of our technique can be useful to improve mutation testing tools by avoiding the application of 13 mutations for 7 mutation targets.

Related papers

Detecting Trivial Mutant Equivalences via Compiler Optimisations
Nikolaos Malevris

IEEE Transactions on Software Engineering

Mutation testing realises the idea of fault-based testing, i.e., using artificial defects to guide the testing process. It is used to evaluate the adequacy of test suites and to guide test case generation. It is a potentially powerful form of testing, but it is well-known that its effectiveness is inhibited by the presence of equivalent mutants. We recently studied Trivial Compiler Equivalence (TCE) as a simple, fast and readily applicable technique for identifying equivalent mutants for C programs. In the present work, we augment our findings with further results for the Java programming language. TCE can remove a large portion of all mutants because they are determined to be either equivalent or duplicates of other mutants. In particular, TCE equivalent mutants account for 7.4% and 5.7% of all C and Java mutants, while duplicated mutants account for a further 21% of all C mutants and 5.4% Java mutants, on average. With respect to a benchmark ground truth suite (of known equivalent mutants), approximately 30% (for C) and 54% (for Java) are TCE equivalent. It is unsurprising that results differ between languages, since mutation characteristics are language-dependent. In the case of Java, our new results suggest that TCE may be particularly effective, finding almost half of all equivalent mutants.

View PDFchevron_right
Muta-Pro: towards the definition of a mutation testing process
Adenilso Da Silva Simão

Journal of the Brazilian Computer Society, 2006

Mutation Testing originated from a classical method for digital circuit testing and today is used at program and specification levels. It can be used either to generate or to assess the quality of test sets. In spite of being very effective in detecting faults, Mutation Testing is usually considered a high cost criterion due to: i) the large number of generated mutants; ii) the time-consuming activity of determining equivalent mutants; and iii) the mutant execution time.

View PDFchevron_right
Employing Dynamic Symbolic Execution for Equivalent Mutant Detection
Ahmed Ghiduk

IEEE Access, 2019

Equivalent mutants (EM) issue is a key challenge in mutation testing. Many methods were applied for detecting and reducing the equivalent mutants. These methods are classified into four classes: equivalent mutant detection, avoiding the generation of equivalent mutants, higher-order equivalent mutants, and suggesting equivalent mutants. Higher-order mutation testing (HOMT) is considered the strongest employed technique in avoiding the generation of equivalent mutants and reduction of their number. In this paper, a combination of HOMT especially second-order mutation testing (SOMT) and dynamic symbolic execution (DSE) techniques are applied for the automatic detection and reduction of the equivalent secondorder mutants. First, SOMT is used to reduce the number of equivalent mutants. Second, DSE technique is applied to classify the SOMs and detect EM. To assess the efficiency of the proposed technique, it is applied to some subject programs and the results of this technique are compared to the manual results and those of related works. The results showed that the proposed algorithm is more effective in detecting and reducing the number of EM. It detects 94% from the equivalent mutants that have manually analyzed. This percentage is a high percentage comparing with previous studies. Besides, the DEM-DSE technique detects 100% of equivalent mutants for 9 of the 14 subject programs. INDEX TERMS Higher-order mutation testing, dynamic symbolic execution, equivalent mutants.

View PDFchevron_right
Mutation analysis using mutant schemata
Jeff Offutt

1993

Abstract Mutation analysis is a powerful technique for assessing and improving the quality of test data used to unit test software. Unfortunately, current automated mutation analysis systems suffer from severe performance problems. This paper presents a new method for performing mutation analysis that uses program schemata to encode all mutants for a program into one metaprogram, which is subsequently compiled and run at speeds substantially higher than achieved by previous interpretive systems.

View PDFchevron_right
Reducing the Cost of Higher-Order Mutation Testing
Moheb Girgis

Arabian Journal for Science and Engineering, 2018

Constructing mutants of order higher than first order is a key step in higher-order mutation testing. The majority of higher-order mutant generation techniques merge two (or more) first-order mutants (FOMs) to build a higher-order mutant. Unfortunately, these techniques suffer from the high cost due to the explosion in the number of higher-order mutants (HOMs). Consequently, developing techniques to find the minimum adequate and effective number of mutants are desired. Earlier work reduced the number of mutants by considering only a subset of mutants, a subset of operators, or selecting specific locations in the program (to be mutated) instead of the whole program. In this paper, we present three new techniques (SCWR, 2E2O, and 2E2OWR) to generate and reduce the overall number of HOMs and the equivalent ones as well. Each technique merges FOMs in more effective and simpler way than the previous techniques to find more effective HOMs. These techniques have been applied on a benchmark of programs, and the results have been compared to the results of some related work such as DiffOp, JudyDiffOp, and Last2First techniques. The results showed that SCWR, 2E2O, and 2E2OWR outperformed the related work and reduced the total number of mutants by 67.9, 16.0, and 60.1% comparing to approximately 50% for the related work and the number of equivalent mutants by 66.9, 79.6, and 65.0% comparing to 25.8 and 36.4% for the related work, respectively. Keywords Mutation testing • Higher-order mutation testing • Mutant generation • First-and higher-order mutants B Ahmed S. Ghiduk

View PDFchevron_right
Towards automating the generation of mutation tests
Nikolaos Malevris

Proceedings of the 5th Workshop on Automation of Software Test - AST '10, 2010

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Towards this direction various automated test data generation tools have been developed. The majority of them aim at branch testing, while a quite limited number aim at a higher level of testing thoroughness such as mutation. In this paper an automated framework that makes a joint use of diverse techniques and tools is introduced in the context of automating mutation based test generation. The motivation behind this work is the use of existing techniques and tools such as symbolic execution and evolutionary testing towards automating the test input generation activity according to the weak mutation testing criterion. The proposed framework integrates existing automated tools for branch testing in order to effectively generate mutation test data. To fulfill this suggestion three automated tools are used for illustration purposes and preliminary results are obtained by applying the proposed framework to a set of java program units indicating the applicability and effectiveness of the proposed approach.

View PDFchevron_right
Detecting Equivalent Mutants by Means of Constraint Systems
Simona Nica

… on Advances in System Testing and …, 2011

Mutation testing has been used along the research community as an efficient method to evaluate the process of software testing, i.e., the quality of the test suite. One major drawback is represented by the equivalent mutant problem. Through this current research we aim to come with a reliable solution to this problem and improve the available test suite pool. We do this by combining the mutation testing procedure together with a constraint satisfaction paradigm and the concept of distinguishing test cases. Mutation testing has been seen, in most of the cases, as a measure for evaluating the quality of a user's test suite. But, also mutation testing can be of great help in the test case generation process. By means of a constraint system we generate test scenarios able to distinguish between two different versions of a program. We start from the hypothesis that when our constraint system is not able to find any solution it might be the case that two equivalent mutants were encountered. The first empirical results, i.e. an increased mutation score, encourage us to further apply the strategy on medium size applications.

View PDFchevron_right
Effectiveness of Mutation Testing Techniques: Reducing Mutation Cost
Bouchaib Falah

The field of mutation testing has been neglected by industry for a long time because of its high cost. The rising research in this field has resulted in the development of several approaches that aim to reduce the cost of testing and to assess the quality of mutants' generation and destruction. Some approaches suggest decreasing the number of operators, while others intend to reduce the execution time. The goal of this paper is to evaluate the effectiveness of three approaches that reduce the mutation testing cost: classlevel, method-level, 10-selective and all operators approach. The experimental part of this research was based on testing seven Java programs using MuClipse tool. The analysis of the resulting findings has proved that the cost of mutation testing can be minimized by selecting a subset of the mutation operators and by adopting the class-level mutation operators for fault detection in java programs.

View PDFchevron_right
Mutation Testing : A Review
pawan kumar chaurasia

Journal of Global Research in Computer Science, 2014

Mutation testing is a fault based testing technique in which mutants are generated in the program and apply different test cases on the mutants. Some mutants are killed and some mutants are alive. On the base of killed and alive mutants, mutant score is calculated. Mutants are categorizing into weak, strong and firm mutants on the cost reduction methods. This paper is used to review the mutation testing and categorize the mutants and focus on cost reduction techniques.

View PDFchevron_right
Using Constraints for Equivalent Mutant Detection
Simona Nica

arXiv preprint arXiv:1207.2234, 2012

View PDFchevron_right

References (27)

  1. Acree, J. A. T. (1980). On mutation. PhD thesis, Georgia Institute of Technology. Anais Estendidos do XI Congresso Brasileiro de Software: Teoria e Prática (CBSoft 2020)
  2. Artifacts (2020). A technique to identify equivalent mutants. At https: //colab.research.google.com/drive/1ts_2m0tW0GLWsVJ_ qhC59V9l8rVPdtsa?usp=sharing.
  3. Baldwin, D. and Sayward, F. (1979). Heuristics for determining equivalence of program mutations. Technical report, DTIC Document.
  4. Budd, T. and Angluin, D. (1982). Two notions of correctness and their relation to testing. Acta Informatica, 18(1):31-45.
  5. de Moura, L. M. and Bjørner, N. (2008). Z3: an efficient SMT solver. In Proceedings of the Tools and Algorithms for the Construction and Analysis of Systems, pages 337-340.
  6. DeMillo, R. A., Lipton, R. J., and Sayward, F. G. (1978). Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34-41.
  7. Fernandes, L., Ribeiro, M., Carvalho, L., Gheyi, R., Mongiovi, M., Santos, A., Caval- canti, A., Ferrari, F., and Maldonado, J. C. (2017). Avoiding useless mutants. In Proceedings of the Generative Programming: Concepts & Experiences, pages 187- 198.
  8. Grün, B. J., Schuler, D., and Zeller, A. (2009). The impact of equivalent mutants. In Proceedings of the International Conference on Software Testing, Verification, and Validation Workshops, pages 192-199.
  9. Guimarães, M., Fernandes, L., Ribeiro, M., d'Amorim, M., and Gheyi, R. (2020). Op- timizing mutation testing by discovering dynamic mutant subsumption relations. In Proceedings of the International Conference on Software Testing, Verification and Va- lidation, page To appear. IEEE.
  10. Harman, M., Hierons, R., and Danicic, S. (2001). The relationship between program dependence and mutation analysis. In Proceedings of the Mutation Testing for the New Century, pages 5-13.
  11. Howden, W. (1982). Weak mutation testing and completeness of test sets. Transactions on Software Engineering, 8(4):371-379.
  12. Jia, Y. and Harman, M. (2009). Higher order mutation testing. Information and Software Technology, 51(10):1379 -1393.
  13. Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. Transactions on Software Engineering, 37(5):649-678.
  14. Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., and Fraser, G. (2014). Are mutants a valid substitute for real faults in software testing? In Proceedings of the Foundations of Software Engineering, pages 654-665.
  15. Kintis, M. and Malevris, N. (2015). MEDIC: A static analysis framework for equivalent mutant identification. Information and Software Technology, 68:1-17.
  16. Kintis, M., Papadakis, M., Jia, Y., Malevris, N., Traon, Y. L., and Harman, M. (2017). De- tecting trivial mutant equivalences via compiler optimisations. Transactions on Soft- ware Engineering, 44(4):308-333.
  17. Ma, Y.-S. and Offutt, J. Description of MuJava's method-level mutation operators. At https://cs.gmu.edu/ ˜offutt/mujava/mutopsMethod.pdf.
  18. Ma, Y.-S., Offutt, J., and Kwon, Y.-R. (2005). MuJava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97-133.
  19. Madeyski, L., Orzeszyna, W., Torkar, R., and Jozala, M. (2014). Overcoming the equiva- lent mutant problem: A systematic literature review and a comparative experiment of second order mutation. Transactions on Software Engineering, 40(1):23-42.
  20. Offutt, A. J. and Lee, S. D. (1994). An empirical evaluation of weak mutation. Transac- tions on Software Engineering, 20(5):337-344.
  21. Offutt, J. (2011). A mutation carol: Past, present and future. Information and Software Technology, 53(10):1098 -1107.
  22. Offutt, J. and Pan, J. (1997). Automatically detecting equivalent mutants and infeasible paths. Software Testing, Verification and Reliability, 7(3):165-192.
  23. Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. L., and Harman, M. (2019). Chap- ter six -mutation testing advances: An analysis and survey. Advances in Computers, 112:275-378.
  24. Petrovic, G. and Ivankovic, M. (2018). State of mutation testing at google. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engi- neering in Practice Track (ICSE-SEIP), pages 163-171.
  25. Pizzoleto, A., Ferrari, F., Offutt, J., Fernandes, L., and Ribeiro, M. (2019). A systema- tic literature review of techniques and metrics to reduce the cost of mutation testing. Journal of Systems and Software, 157.
  26. Schuler, D. and Zeller, A. (2013). Covering and uncovering equivalent mutants. Software Testing, Verification and Reliability, 23(5):353-374.
  27. Winterer, D., Zhang, C., and Su, Z. (2020). Validating SMT solvers via semantic fusion. In Proceedings of the Programming Language Design and Implementation, pages 718- 730.

Related papers

Optimizing Mutation Testing by Discovering Dynamic Mutant Subsumption Relations
Marcio Augusto Guimarães

2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST)

One recent promising direction on reducing costs of mutation analysis is to identify redundant mutations, i.e., mutations that are subsumed by some other mutations. Previous works found out redundant mutants manually through the truth table. Although the idea is promising, it can only be applied for logical and relational operators. In this paper, we propose an approach to discover redundancy in mutations through dynamic subsumption relations among mutants. We focus on subsumption relations among mutations of an expression or statement, named here as "mutation target." By focusing on targets and relying on automatic test generation tools, we define subsumption relations for dozens of mutation targets in which the MUJAVA tool can apply mutations. We then implemented these relations in a tool, named MUJAVA-M, that generates a reduced set of mutants for each target, avoiding redundant mutants. We evaluated MUJAVA and MUJAVA-M using classes of five open-source projects. As results, we analyze 2,341 occurrences of 32 mutation targets in 168 classes. MUJAVA-M generates less mutants (on average 64.43% less) with 100% of effectiveness in 20 out of 32 targets and more than 95% in 29 out of 32 mutation targets. MUJAVA-M also reduced the time to execute the test suites against the mutants in 52.53% on average, considering the full mutation analysis process.

View PDFchevron_right
Quantitative Metrics for Mutation Testing
Amani Ayad

Proceedings of the 14th International Conference on Software Technologies, 2019

View PDFchevron_right
Identifying method-level mutation subsumption relations using Z3
Beatriz Souza

Information and Software Technology, 2021

Context: Mutation analysis is a popular but costly approach to assess the quality of test suites. One recent promising direction in reducing costs of mutation analysis is to identify redundant mutations, i.e., mutations that are subsumed by some other mutations. A previous approach found redundant mutants manually through truth tables but it cannot be applied to all mutations. Another work derives them using automatic test suite generators but it is a time consuming task to generate mutants and tests, and to execute tests. Objective: This article proposes an approach to discover redundant mutants by proving subsumption relations among method-level mutation operators using weak mutation testing. Method: We conceive and encode a theory of subsumption relations in the Z3 theorem prover for 37 mutation targets (mutations of an expression or statement). Results: We automatically identify and prove a number of subsumption relations using Z3, and reduce the number of mutations in a number of mutation targets. To evaluate our approach, we modified MuJava to include the results of 24 mutation targets and evaluate our approach in 125 classes of 5 large open source popular projects used in prior work. Our approach correctly discards mutations in 75.93% of the cases, and reduces the number of mutations by 71.38%. Conclusions: Our approach offers a good balance between the effort required to derive subsumption relations and the effectiveness for the targets considered in our evaluation in the context of strong mutation testing.

View PDFchevron_right
Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation
Richard Torkar, Lech Madeyski

IEEE Transactions on Software Engineering, 2000

View PDFchevron_right
Efficiently Finding Useless Mutants
Beatriz Souza

2021

Mutation analysis is a popular but costly approach to assess the quality of test suites. Equivalent and redundant mutants contribute to increase costs and are harmful to the design of test suites. We propose a lightweight technique to identify equivalent and redundant mutants based on theorem proving with Z3 in the context of weak mutation testing. The experiments reveal that our technique detects all equivalent mutants detected by TCE and we have an average reduction of 72.52% of mutants, when considering entire programs. We also apply our technique on HOMs. When considering both FOMs and HOMs, 91% of the mutations could be discarded on average. The results found by our approach may help to make mutation testing less expensive and more accurate.

View PDFchevron_right

Related topics

  • Computer Science
  • Chemistry
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved