A Deterministic T-Way Strategy for Test Data Minimization

Journal of Advances in Information Technology

Exhaustive testing is a type of testing that test all possible combination of parameter values to make sure the product is free from any possible faults and errors. However, employing exhaustive testing would be impossible due to the factors that includes time, labor, cost, and resource constraints. Thus, combinatorial t-way testing was introduced to complement exhaustive testing problem. The t-way testing generates small test list considering that each test case can cover the greatest number of interaction tuples based on the t-way coverage. Many works were done in the field of combinatorial test generation to generate an optimized test suite based on a given t-way interaction strength, since t-way testing is exceedingly complicated (NPhard). The t-way techniques make use of "don't care" values when generating optimal test suite, but most of research adopt the values without a critical look into the consequences around them that lead to a higher number of test cases generated. As such, this paper presents the influence of "don't care" values in the field of t-way testing, we also show that this line of research is threatening to lead the area away from optimization rigor. A case study is given as our practical example, we applied each t-way techniques up to two times to show the impact of "don't care" values. Moreover, we suggest to the proper way of adopting "don't care" values for t-way testing strategies. Finally, our suggestion can be applying when implementing t-way strategy consciously adopt the adjustment to reduce test suite size.

Combinatorial t-way technique is efficient in generating test data and addressing the problem of combinatorial explosion. When constructing a test case, numerous literatures classified t-way strategies into two basic approaches as either One-test-at-a-time approach (OTAT) and One-parameter-at-a-time approach (OPAT). At least three major challenge groups can be encountered when creating test cases. The first one is provision of parameters seeding support that will improve the software quality. The second involves automatically obtaining data regarding parameter constraints and identifying interactions between system components. The last one is the execution speed and the test suite size. However, in all the existing OPAT t-way strategies, given that the system is loaded with this information, testing present-day software systems is made difficult or impossible. This study presents an effective combinatorial t-way test case generation strategy named Seeding and Constraint support in In-Parameter-Order Generalized (SCIPOG), to develop an improved paired testing approach. However, the study examines the present state-of-the-art and compares several OPAT strategies found in the literature. Moreover, experiments are discussed as part of this process to demonstrate the correctness of the implementation. When statistically analyzing the findings, two non-parametric tests-the Wilcoxon Rank and Friedman testswere run. SCIPOG, however, produced competitive results. Finally, SCIPOG showed the efficiency of the two proposed methods, which are seeding and constraint support in IPOG strategy.

Lecture Notes in Computer Science, 2008

Software testing is an integral part of software engineering. Lack of testing often leads to disastrous consequences including loss of data, fortunes, and even lives. In order to ensure software reliability, many combinations of possible input parameters, hardware/software environments, and system configurations need to be tested and verified against for conformance. Due to costing factors as well as time to market constraints, considering all exhaustive test possibilities would be infeasible (i.e. due to combinatorial explosion problem). Earlier work suggests that pairwise sampling strategy (i.e. based on twoway parameter interaction) can be effective. Building and complementing earlier work, this paper discusses an efficient pairwise test data generation strategy, called IRPS. In doing so, IRPS is compared against existing strategies including AETG and its variations, IPO, SA, GA, ACA, and All Pairs. Empirical results demonstrate that IRPS strategy, in most cases, outperformed other strategies as far as the number of test data generated within reasonable time.

2006

The primary purpose of Software Testing Process and Evaluation (STP& E) is to reduce risk. All testing provides insight and helps identify "unknown-unknowns". This paper describes STP with Assured Confidence techniques. The uncertainty problem is an important issue in the computer industry today and testing is still the main technique for quality assurance. There is a need to ensure that the software is reasonably safe from severe faults after testing. When faced with financial or a schedule constraints, testing is usually cut horizontally attempting to cover as many different test requirements at the expense of depth. We have reached a point where we must test smarter and apply Statistical-Risk-Based Test with Assured Confidence (SRBTAC) management procedure. We need to pick the right assessment tools to make vertical cuts in our test strategies. Any given test has a low probability of detecting a problem, but all the possibilities create a too-high probability of defects...

2008

This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference point is provided by hand generated test vectors constructed during development to establish minimum acceptance criteria.

2000

Covering as much as 40 to 50 percent of the development costs and resources, software testing is an integral part of the software development lifecycle. Despite its importance, current software testing practice lacks automation and is still primarily based on highly manual processes from the generation of test cases (i.e. from the specifications documents) up to the actual execution of the test. These manually generated tests are sometimes executed using ad hoc approach, typically requiring the construction of a test driver for the particular application under test. The construction of a test driver is tedious, error prone, and cumbersome process, as it puts extra burden to test engineers. In addition, test engineers are also under pressure to test increasing lines of code in order to meet market demands and deadlines for more software functionalities. To attain the required level of quality and maintain acceptable test coverage, test engineers need to consider a significantly large number of test cases. Many combinations of possible input parameters, hardware/software environments, and system conditions need to be tested and verified against for conformance based on the system's specification. Often, this results into combinatorial explosion of test cases. Addressing some of the aforementioned issues, this paper describes a new automated and combinatorial software testing tool, called JTst, based on the use of Java technology. The main novel features of JTst are the fact that it permits combinatorial test data generation and automated concurrent execution, enabling higher product quality at lower testing costs.

Asian Journal of Computer Science and Technology, 2018

During software development, testing and re-testing occurs frequently to ensure that the software is working correctly before and after modifications. To carry out an effective testing process a test suite is created and executed to detect the faults in the existing code as well as in the modified code. The manual approach of test suite creation and execution is time consuming and labour intensive task as compared to automatically generated test data or test suite. The automatic test data generation is supposed to be an effective way, but a lot of redundant test cases are generated that increase the time, effort and cost of testing. Therefore, test suite minimization techniques are used to further minimize or reduce the number of test cases by selecting a subset from an initially random and large test suite to test the code before as well as after modification. In this study, a comprehensive analysis of the different test suite minimization techniques is presented in order to extend the existing studies and to propose new ideas in this direction.

2014

Software testing relates to the process of accessing the functionality of a program. To ensure conformance, test engineers often generate a set of test cases to validate against the user requirements. When dealing with large line of codes (LOCs), there are potentially issues of redundancy as new test cases may be added and old test cases may be deleted during the whole testing process. To address redundancy issues, many useful strategies (e.g. HGS, GE, and GRE) have been developed in the literature. These strategies often put focus on getting the most minimum test suite size but give poor emphasis on test prioritization (i.e. ordering of tests). Here, as most testing activities happen toward the end of software development, testers are often forced to consider partial test suite, that is, to be in line with the project deadline. In this manner, some impactful defects may be missed owing to the need to accommodate deadline shift from earlier development activities. In order to address these issues, this paper highlights our ongoing work on the development of a novel test redundancy reduction strategy based Late Acceptance Hill Climbing, called (LAHCS). LAHCS is the first known strategy that adopts Late Acceptance Hill Climbing Algorithm for test redundancy reduction and prioritization.

Regression testing is a critical activity which occurs during the maintenance stage of the software lifecycle. However, it requires large amounts of test cases to assure the attainment of a certain degree of quality. As a result, test suite sizes may grow significantly. To address this issue, Test Suite Reduction techniques have been proposed. However, suite size reduction may lead to significant loss of fault detection efficacy. To deal with this problem, a greedy algorithm is presented in this paper. This algorithm attempts to select a test case which satisfies the maximum number of testing requirements while having minimum overlap in requirements coverage with other test cases. In order to evaluate the proposed algorithm, experiments have been conducted on the Siemens suite and the Space program. The results demonstrate the effectiveness of the proposed algorithm by retaining the fault detection capability of the suites while achieving significant suite size reduction.