Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Background
Background and Definition
Conclusion
References

AN ANALYSIS OF TEST SUITE MINIMIZATION TECHNIQUES

Profile image of IJESRT JournalIJESRT Journal
https://doi.org/10.5281/ZENODO.165632
visibility

…

description

9 pages

link

1 file

Abstract

In software development, testing is widely used by developers to reveal faults that cause failures and improve the quality of the software by omitting or removing the detected faults. Test suites for testing and validating software programs can contain large numbers of test cases to execute various parts of the software program code to check for defects and code compliance. Test suite sizes may grow significantly with subsequent modifications to the software over time that causes redundancy in test suites. Due to time and resource constraints retesting the modified software every time, it is advantageous to develop techniques that manage test suite sizes by periodically removing duplicate test cases, in turn, can improve the maintenance efforts required for regression testing. The primary objective of test suite reduction is to effectively improve the testing process while maintaining Fault Detection Effectiveness and fulfilling all the testing requirements as well. Researchers have proposed some TSR heuristics based on different variations of a greedy algorithm. This paper presents a comparative study of test suite reduction techniques based on some parameters.

Figures (3)
See Table 1 that shows an example of the coverage information of test cases in a test suite {T1, T2, T3, T4, and T5}. The symbol (x) represents satisfaction of a requirement by a test case. Here we find that a subset {T1, TS, T4} of the suite cover all the requirements {R1, R2, R3, R4, R5, R6, R7}, whereas test cases T2 and T5 become duplicate due to the requirements covered by them are fulfilled by the other three test cases T1, T3, and T5. The main objective of test suite reduction is to optimize the regression testing process while maintaining Fault Detection Effectiveness (FDE) [5]. At the time of minimization process, it may be useful to choose the test cases that are likely to reveal faults instead of including more test cases in the reduced suite. For a particular program, a test selection criterion translates into a requirement set, whose satisfaction provides that how much this fulfill the specified criterion. Analysis Parameter This section defines some set of analysis parameters for analyzing existing algorithms for test suite reduction. We have found these analysis parameters from the referenced papers which work as a base to carry out the comparison process.
See Table 1 that shows an example of the coverage information of test cases in a test suite {T1, T2, T3, T4, and T5}. The symbol (x) represents satisfaction of a requirement by a test case. Here we find that a subset {T1, TS, T4} of the suite cover all the requirements {R1, R2, R3, R4, R5, R6, R7}, whereas test cases T2 and T5 become duplicate due to the requirements covered by them are fulfilled by the other three test cases T1, T3, and T5. The main objective of test suite reduction is to optimize the regression testing process while maintaining Fault Detection Effectiveness (FDE) [5]. At the time of minimization process, it may be useful to choose the test cases that are likely to reveal faults instead of including more test cases in the reduced suite. For a particular program, a test selection criterion translates into a requirement set, whose satisfaction provides that how much this fulfill the specified criterion. Analysis Parameter This section defines some set of analysis parameters for analyzing existing algorithms for test suite reduction. We have found these analysis parameters from the referenced papers which work as a base to carry out the comparison process.
Table 3. Comparative Analysis of TSM Techniques http: // www.ijesrt.com
Table 3. Comparative Analysis of TSM Techniques http: // www.ijesrt.com

Related papers

An Efficient Heuristic Based Test Suite Minimization Approach
DIBYA BORA

Indian Journal of Science and Technology, 2017

Objectives: Development of an efficient test suite minimization approach in order to reduce the size of a previously acquired test suite and produce a new representative suite which will guarantee the same requirement coverage that was achieved before minimization for an effective and efficient regression testing. Method: Test suite minimizations techniques try to reduce the size and redundancy of test suite by removing certain test cases since requirement covered by them are already covered by other test cases. But, it has been found that the acquired test cases after minimization severely lacks ability to achieve the desirable code coverage because the minimization was done based on a single test adequacy criteria. In this paper, we propose an efficient heuristic based test suite minimization algorithm which will reduce the size of the test suites with respective to multiple test adequacy criterions in order to preserve the fault detection effectiveness and code coverage characteristics of the final test suite. Findings: Our experimental results indicate that a significant percentage of reduction in the test suite size is achieved when the minimization is performed with respect to multiple test adequacy criterions. Our approach is unique compared to the existing approaches in the sense that, we carried out minimization based on multiple test adequacy criterions while most of the existing approaches usually take one or two criterions into consideration. The proposed approach is evaluated based on two well known software testing metrics; one indicate the percentage of reduction in test suite size and the second one indicate the percentage of code coverage achieved by the minimized test suite. Our experimental results indicate that a significant percentage of reduction in the size as well as significant code coverage characteristics is achieved when the minimization is done according to the proposed approach. Improvements: The important contribution of this study is that, it presents a novel and efficient test suite minimization technique that optimizes the test suite size based on multiple adequacy criterions.

View PDFchevron_right
A New Similarity-Based Greedy Approach for Generating Effective Test Suite
Shilpi Singh

International Journal of Intelligent Engineering and Systems, 2018

Software regression testing is one of the most critical phases of software development life cycle, used by developers with the intent of detecting new faults to validate modified software prior to delivery to the customer. To validate updated features, new test cases are generated by the testers which increment the test suite size automatically. The resulting test suite may contain obsolete, redundant, and ambiguous test cases. Therefore, there is a strong requirement of an intelligent testing approach to reduce the test suite size by removing those unessential test cases economically. This paper proposed an interesting approach, which involves the combination of regression testing techniques: minimization, and prioritization both. The main focus is on multiple regression activities with multiple criteria rather than using only single activity to produce an optimal solution. In this paper clustering approach is also considered, which could simplify and enhance the minimization and p...

View PDFchevron_right
A new algorithm to Test Suite Reduction based on cluster analysis
Alireza Khalilian, Yalda Fazlalizadeh

2009 2nd IEEE International Conference on Computer Science and Information Technology, 2009

Regression testing is a significant activity which occurs during the maintenance level of a software lifecycle. However, it requires a large amount of test cases to test new or modified parts of the software. To address the issue, Test Suite Reduction techniques have been presented. An appropriate technique should generate a minimized test suite which exercises different execution paths within a program while retaining the fault detection capability of the suite admissible. To achieve this, a heuristic algorithm is proposed in this paper. The new algorithm clusters test cases based on the similarity of their execution profiles and sample some representatives to form the reduced test suite. The results of applying the new algorithm to the Siemens suite and comparing to the H algorithm manifest interesting insights into the effectiveness of the proposed algorithm.

View PDFchevron_right
Cost-Aware Test Suite Minimization Approach Using TBAT Optimization Algorithm for Software Testing
Eswara Reddy

Traditional way of optimizing regression testing cost is to reduce subsets of test cases from a test suite without compromising the test requirement. In order to reduce the test suite, researchers have presented various test-suite reduction techniques using coverage metrics and greedy search algorithms. Besides greedy algorithms, optimization-based algorithms have played a major role in test suite reduction. Accordingly, we developed a new optimization algorithm called, TBAT algorithm to handle the diversity problem in generating new solutions while finding the optimal test cases. Here, a fitness function is developed to select the test cases optimally through the TBAT algorithm using two constraints, satisfying the entire test requirement and minimizing the cost measure. The proposed TBAT algorithm is experimented with five programs from SIR using four different evaluation metrics. The empirical study on the performance of the TBAT algorithm is analyzed with various parameters and ...

View PDFchevron_right
A Code Coverage-Based Test Suite Reduction and Prioritization Framework
Saif Ur Rehman Khan

Software testing is extensively used to ensure the development of a quality software system. The test suite size tends to increase by including new test cases due to software evolution. Consequently, the entire test suite cannot be executed considering budget and time limitations. Researchers have examined test suite reduction and prioritization techniques to address the test suite size problem. However, combination of these techniques can be useful for various regression testing situations. In this paper, we present a new code coverage-based test suite reduction and prioritization framework called TestOptimizer. The framework performs a suitable combination of TestFilter and St-Total techniques to determine optimal test cases, keeping in view of time restrictions. The performance of the proposed framework has been assessed using a case study. Results show that TestOptimizer can be beneficial to solve the test suite size problem within time constraints and has a profound impact on the required cost and effort of regression testing.

View PDFchevron_right
Test Suite Minimization
Javairia Khan

Programming Testing is a basic piece of programming improvement. Programming testing and retesting happens ceaselessly amid the product advancement life cycle. As programming develops and advances, new experiments are produced furthermore, added to a test suite to practice the most recent changes to the product. More than a few forms of the advancement of the product, some experiments in the test suite may wind up noticeably excess as for the testing prerequisites for which they were produced since these prerequisites are currently likewise fulfilled by other experiments in the suite that were added to cover changes in the later forms of programming. Because of time and asset imperatives for retesting the product each time it is adjusted, it is critical to create systems that keep test suite sizes reasonable by occasionally expelling repetitive experiments. This procedure is called test suite minimization. Test suite diminishment (TSR) is to discover a subset of the test suite containing an insignificant number of experiments that can fulfill all test prerequisites. Test suite decrease systems endeavor to evacuate excess experiments Programming testing and retesting happens ceaselessly amid the delicate product improvement lifecycle to recognize blunders as ahead of schedule as could reasonably be expected and to guarantee that progressions to existing programming don't break the delicate product. Test suites once created are reused and refreshed every now and again as the product develops. Therefore, some experiments in the test suite may end up noticeably excess as the product is adjusted after some time since the necessities secured by them are additionally secured by other experiments. Because of the asset and time limitations for re-executing substantial test suites, it is imperative to create methods to limit accessible test suites by evacuating repetitive experiments. By and large, the test suite minimization issue is NP finished. The fundamental goal of test suite minimization, purported test suite improvement or test suite diminished, is to reduce the quantity of experiments inside a test suite while keeping up

View PDFchevron_right
Optimizing Regression Test Suite Reduction
Sivaji U

Springer, 2018

Software reliability is one of the essential features in the system quality. The purpose of the regression testing is to establish quality of the modified software before the software deployment. The main issue in this regression model is how to selecting a subset of test cases from the large test suite in the modified software version. Conventional regression subset-selection models have addressed this issue on a small set of test suites or limited test cases. In this paper, we have studied the background work of different hybrid regression models and its limitations on the standard test suite. For different software applications the proposed methodology is applied to find various combinations of faults randomly injected in various combinations of code lines as specified. We use selenium builder to test the software applications and run it in junit eclipse IDE to find faults. The tested output is interpreted with java randomly induced faults and the system of methodology is evaluated in terms of APFD and APFD box plot.

View PDFchevron_right
Frequent Segment Clustering of Test cases for Test Suite Reduction
Narendra Kumar Rao

Execution profiles are indicators for code coverage of program; this has been demonstrated by researchers on a large scale through their contributions on the same. Test Suite reduction is a feature which achieves code coverage with minimum number of test cases ensuring that all code items have been tested. It is a Non-deterministic Polynomial-time Complete (NP-Complete) problem. Few approaches like Greedy approach, Harrold,Soffa and Gupta(HGS) approach have been used in literature which are good approaches. Current work achieves similar milestones with reduced test cases as well. This paper presents Maximal frequency item set clustering and sequencing of similar test cases, residue code requirements based test case reduction and modification based test case selection. In the current work, few interesting results were found, where in similar program trace test cases were greatly reduced ensuring high code coverage percentage during testing. Fault detection can be tuned by selecting test cases from similar groups.

View PDFchevron_right
Test Case Reduction Techniques - Survey
Dima Suleiman

International Journal of Advanced Computer Science and Applications, 2016

Regression testing is considered to be the most expensive phase in software testing. Therefore, regression testing reduction eliminates the redundant test cases in the regression testing suite and saves cost of this phase. In order to validate the correctness of the new version software project that resulted from maintenance phase, Regression testing reruns the regression testing suite to ensure that the new version. Several techniques are used to deal with the problem of regression testing reduction. This research is going to classify these techniques regression testing reduction problem.

View PDFchevron_right
An Efficient Approach for Test Suite Reduction Using K-Means Clustering
Ahmad A Saifan

2018

Software testing is the primary approach that is used to test and evaluate software under development. The main goal of testing is to find defects before customers find them out. It is very costly. Therefore, reducing the cost of the test is a big challenge. This paper aims at reducing the cost of the test by eliminating the redundant test cases. Our methodology begins with generating the test cases randomly. The Procedural Language/Structured Query Language (PL/SQL) tool is used to generate test cases from the payroll system database functions. The SPSS software package is used to apply the K-means Clustering algorithm to reduce the test cases. The results reveal that the proposed approach significantly reduces the number of test cases from 776 to 240 while keeping the same coverage.

View PDFchevron_right

References (31)

  1. Gregg Rothermel, Mary Jean Harrold, Jeffery von Ronne, and Christie Hong, "Empirical Studies of Test Suite Reduction", Journal of Software Testing, Verification, and Reliability. 12(4):219-249, 2002.
  2. D. Binkley, "Semantics guided regression test cost reduction", IEEE Trans. Softw. Eng. 23 (8) (1997) 498-516.
  3. H. Zhong, L. Zhang, and H. Mei, "An experimental study of four typical test suite reduction techniques", Inform. Softw. Technol. 50 (6) (2008) 534-546.
  4. G. Rothermal, R. H. Untch, C. Chu, and M. J. Harrold, "Prioritizing test cases for regression testing", IEEE Transactions on software Engineering, 27 (10)(2001) 929-948Tavel, P. 2007 Modelling and Simulation Design. AK Peters Ltd
  5. D. Jeffrey and N. Gupta, "Improving fault detection capability by selectively retaining test cases during test suite reduction", IEEE Transactions on software Engineering, 33(2):108-123, 2007.
  6. R. A. DeMillo and A. J. Offutt, "Constraint-based automatic test data generation", IEEE Trans. on Soft. Eng.", 17(9):900-910, 1991.
  7. Gupta, Rajiv, and Mary Lou Soffa, "Employing static information in the generation of test case", Software Testing, Verification and Reliability 3.1 (1993): 29-48.
  8. T.Y. Chen, and M.F. Lau, "A Simulation Study on Some Heuristics for Test Suite Reduction", Journal of Information and Software Technology, vol. 40, issue 13, pp. 777-787, 1998.
  9. J.V. Ronne, "Test Suite Minimization: An Empirical Investigation", Bachelors Thesis, June 1999. Retrieved from URL:http://www.ics.uci.edu/jronne/pubs/jvronne uhc-thesis.pdf
  10. S. Elbaum, A. Malishevsky, and G. Rothermel, "Test case prioritization: a family of empirical studies", IEEE Transactions on Software Engineering28 (2) (2002) 159-182.
  11. T. Graves, M. Harrold, J. Kim, A. Porter, and G. Rothermel, "An empirical study of regression test selection techniques", ACM Transactions on Software Engineering and Methodology (TOSEM) 10 (2)(2001) 184-208.
  12. Zhong, Hao, Lu Zhang, and Hong Mei, "An experimental comparison of four test suite reduction techniques", Proceedings of the 28th international conference on Software engineering.ACM, 2006.
  13. J.W. Lin, and C.Y. Huang, "Analysis of Test Suite Reduction with Enhanced Tie-Breaking Techniques", Journal of Information and Software Technology, vol. 51, issue 4, pp. 679-690, 2009.
  14. G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong, "An empirical study of the effects of minimization on the fault detection capabilities of test suites", International Conference on Software Maintenance, pages 34-43, November 1998.
  15. F.I. Vokolos, and P.G. Frankl, "Empirical evaluation of the textual differencing regression testing technique", in Proceedings of 14thInternational Conference on Software Maintenance, 1998, pp.44-53.
  16. C. Sharma, S. Singh, "Mechanism for identification of duplicate test cases", In Recent Advances and Innovations in Engineering (ICRAIE), 2014 (pp. 1-5). IEEE.
  17. J.A. Jones, and M.J. Harrold, "Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage", IEEE Transactions on Software Engineering, vol. 29, issue 3, pp. 195-209, 2003.
  18. W.E. Wong, J.R. Horgan, S. London, and A.P. Mathur "Effect of Test Set Minimization on Fault Detection Effectiveness of the All-Uses Criterion", Proceedings of the 17th International Conference on Software Engineering (ICSE'94), pp. 41-50, 1994.
  19. Khan, Saif Ur Rehman, et al. "An Analysis of the Code Coverage-based Greedy Algorithms for Test Suite Reduction". The Second International Conference on Informatics Engineering & Information Science (ICIEIS2013),The Society of Digital Information and Wireless Communication, 2013.
  20. Chvatal, Vasek., "A greedy heuristic for the set-covering problem." Mathematics of operations research 4.3 (1979): 233-235.
  21. S. Yoo, and M. Harman, "Regression testing minimization, selection and prioritization: a survey", Softw. Test., Verif. Reliab. 22 (2) (2012) 67-120.
  22. M.J.Harrold, R. Gupta, and M .L. Soffa, "A methodology for controlling the size of a test suite".ACM Transactions on Software Engineering and Methodology, 2 (3) (1993) 270-285.
  23. T.Y. Chen ,and M.F. Lau, "A New Heuristic for Test Suite Reduction", Journal of Information and Software Technology, vol. 40, issues 5-6, pp. 347-354, 1998.
  24. J. J. Chilenski, and S. P. Miller, "Applicability of modified condition/decision coverage to software testing. Softw. Eng. Journal, 9(5):193-200, 1994.
  25. S.U.R. Khan, A. Nadeem, and A. Awais, "TestFilter: A Statement Coverage based Test Case Reduction Technique", Proceeding of 10th IEEE International Multitopic Conference (INMIC'06), pp. 275-280, 2006.
  26. H. K. N. Leung and L. White, "Insight into regression testing", In Proceedings of Interntional Conference on Software Maintenance (ICSM 1989). pages 60{69. IEEE Computer Society Press, October 1989.
  27. Gotlieb, Arnaud, and Dusica Marijan, "FLOWER: optimal test suite reduction as a network maximum flow", Proceedings of the 2014 International Symposium on Software Testing and Analysis, ACM, 2014.
  28. Campos, Juan, and Rui Abreu, "Leveraging a Constraint Solver for Minimizing Test Suites, "Quality Software (QSIC), 2013 13th International Conference" on IEEE, 2013.
  29. Khalilian, Alireza, and SaeedParsa. "Bi-criteria test suite reduction by cluster analysis of execution profiles", Advances in Software Engineering Techniques, Springer Berlin Heidelberg, 2012, 243-256.
  30. Usaola, Macario Polo, Pedro Reales Mateo, and Beatriz Pérez Lamancha, "Reduction of test suites using mutation", Fundamental Approaches to Software Engineering, Springer Berlin Heidelberg, 2012, 425-438.
  31. S. Tallam, and N. Gupta "A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization", ACM SIGSOFT Software Engineering Notes, vol. 31, issue 1, pp. 35-42, 2006.

Related papers

On the optimization approach towards test suite minimization
saeed parsa

Regression testing is a critical activity which occurs during the maintenance stage of the software lifecycle. However, it requires large amounts of test cases to assure the attainment of a certain degree of quality. As a result, test suite sizes may grow significantly. To address this issue, Test Suite Reduction techniques have been proposed. However, suite size reduction may lead to significant loss of fault detection efficacy. To deal with this problem, a greedy algorithm is presented in this paper. This algorithm attempts to select a test case which satisfies the maximum number of testing requirements while having minimum overlap in requirements coverage with other test cases. In order to evaluate the proposed algorithm, experiments have been conducted on the Siemens suite and the Space program. The results demonstrate the effectiveness of the proposed algorithm by retaining the fault detection capability of the suites while achieving significant suite size reduction.

View PDFchevron_right
An Analysis of the Code Coverage-based Greedy Algorithms for Test Suite Reduction
SDIWC Organization

Software testing is widely used by development organizations to improve the quality and reliability of a System under Test (SUT). However, testing requires high amount of development resources due to many factors. One important factor is the high cost of executing all generated test cases. Test Suite Reduction (TSR) heuristics focus on finding a minimized set of test cases without compromising on fault detection capability. In addition, TSR can be highly useful in a testing scenario which needs to be performed using actual hardware. Ultimately, TSR helps the development organizations to reduce the amount of estimated development resources. Researchers have proposed a number of TSR heuristics based on different variations of greedy algorithm. In this paper, we present a comparison of proposed code coveragebased greedy algorithms for TSR based on a devised set of analysis parameters. We conclude that most of the greedy algorithm based TSR heuristics use finegranular coverage criteria and focus on greedily solving the single-objective based TSR optimization problem.

View PDFchevron_right
A Sound Assessment of Test Suite Minimization Techniques
Fayaz Ahmad Khan

Asian Journal of Computer Science and Technology, 2018

During software development, testing and re-testing occurs frequently to ensure that the software is working correctly before and after modifications. To carry out an effective testing process a test suite is created and executed to detect the faults in the existing code as well as in the modified code. The manual approach of test suite creation and execution is time consuming and labour intensive task as compared to automatically generated test data or test suite. The automatic test data generation is supposed to be an effective way, but a lot of redundant test cases are generated that increase the time, effort and cost of testing. Therefore, test suite minimization techniques are used to further minimize or reduce the number of test cases by selecting a subset from an initially random and large test suite to test the code before as well as after modification. In this study, a comprehensive analysis of the different test suite minimization techniques is presented in order to extend the existing studies and to propose new ideas in this direction.

View PDFchevron_right
Importance of an Effective Test Suite Minimization Technique in Software Testing
Fayaz Ahmad Khan

International Journal of Recent Technology and Engineering (IJRTE), 2019

Software testing is an important activity for developing a quality end product. But it is expensive as it requires both time and cost to perform it during development phase and after its delivery. Therefore, it is required to develop some techniques in this direction in order to make software testing a desirable as well as an affordable activity. Hence, in this study we have proposed an effective method that minimizes the size and redundancy from an automated random generated test suite using data clustering technique. The proposed technique compared to other existing techniques achieves a substantial amount of reduction in the size of the test suite without affecting the fault detection capability of the test suite.

View PDFchevron_right
A survey on Test Suite Reduction frameworks and tools
Saif Ur Rehman Khan

International Journal of Information Management, 2016

Software testing is a widely accepted practice that ensures the quality of a System under Test (SUT). However, the gradual increase of the test suite size demands high portion of testing budget and time. Test Suite Reduction (TSR) is considered a potential approach to deal with the test suite size problem. Moreover, a complete automation support is highly recommended for software testing to adequately meet the challenges of a resource constrained testing environment. Several TSR frameworks and tools have been proposed to efficiently address the test-suite size problem. The main objective of the paper is to comprehensively review the state-of-the-art TSR frameworks to highlights their strengths and weaknesses. Furthermore, the paper focuses on devising a detailed thematic taxonomy to classify existing literature that helps in understanding the underlying issues and proof of concept. Moreover, the paper investigates critical aspects and related features of TSR frameworks and tools based on a set of defined parameters. We also rigorously elaborated various testing domains and approaches followed by the extant TSR frameworks. The results reveal that majority of TSR frameworks focused on randomized unit testing, and a considerable number of frameworks lacks in supporting multi-objective optimization problems. Moreover, there is no generalized framework, effective for testing applications developed in any programming domain. Conversely, Integer Linear Programming (ILP) based TSR frameworks provide an optimal solution for multi-objective optimization problems and improve execution time by running multiple ILP in parallel. The study concludes with new insights and provides an unbiased view of the state-of-the-art TSR frameworks. Finally, we present potential research issues for further investigation to anticipate efficient TSR frameworks.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved