Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Figures
Test-Impact Analysis T Test-Impact
Test-Impact Analysis
Test-Gap Analysis
Limitations
Conclusion
References
All Topics
Computer Science
Software Engineering

Towards change-driven testing

Profile image of Viktória FördősViktória Fördős

2017, Proceedings of the 16th ACM SIGPLAN International Workshop on Erlang

https://doi.org/10.1145/3123569.3123575
visibility

…

description

14 pages

link

1 file

Abstract

Engineering complex business critical systems that should never stop or fail is very much challenging. In Klarna, we tackle this challenge day by day. The secret sauce, which enables us to ensure the highest software quality, is the thorough validation process. However, one must pay the price of such validation process, which is essentially time. In this paper we present our initial work on fast tracking the validation of code changes without compromising software quality and give a preliminary evaluation on our technique.

Key takeaways
sparkles

AI

  1. Change-Driven Testing identifies 90% of mistakes in 2% of test suite runtime.
  2. The text introduces methods to enhance testing efficiency and effectiveness simultaneously.
  3. Test-Impact Analysis (TIA) finds impacted tests, ensuring fast feedback on code changes.
  4. Test-Gap Analysis (TGA) reveals untested changes, highlighting areas for improvement in testing.
  5. Large amounts of untested code lead to increased field bugs and inefficient testing processes.
Figures (8)
Fig. 1 Test intelligence: combining readily available data from various data sources in the software development process to automatically answer questions about testing
Fig. 1 Test intelligence: combining readily available data from various data sources in the software development process to automatically answer questions about testing
Fig. 2 Treemaps that show data about the software system Teamscale from the version-control system, the ticket system, and profiling test execution. (a) Source code as a Treemap. Each box represents one method. The size of the box is proportional to the lines of code in that method. (b) Code changes since a fixed baseline. New methods are red, changed methods are yellow, and methods that remain unchanged are gray. (c) Code changes for ticket TS- 15717. New methods are red and changed methods are yellow. (d) Test coverage on method-level. Methods that have been executed are green, all other methods are gray
Fig. 2 Treemaps that show data about the software system Teamscale from the version-control system, the ticket system, and profiling test execution. (a) Source code as a Treemap. Each box represents one method. The size of the box is proportional to the lines of code in that method. (b) Code changes since a fixed baseline. New methods are red, changed methods are yellow, and methods that remain unchanged are gray. (c) Code changes for ticket TS- 15717. New methods are red and changed methods are yellow. (d) Test coverage on method-level. Methods that have been executed are green, all other methods are gray
Fig. 4 Process of the Test-Impact Analysis (TIA). Given a set of changes, TIA selects the impacted tests and orders them by their likelihood to find a new mistake in the changes
Fig. 4 Process of the Test-Impact Analysis (TIA). Given a set of changes, TIA selects the impacted tests and orders them by their likelihood to find a new mistake in the changes
To determine test gaps, TGA considers the chronological order of changes and test runs: New changes invalidate any previous test coverage of the changed code and open new test gaps. With subsequent testing, new coverage is recorded, which closes respective test gaps on earlier changes. Consequently, TGA can give us an update on our test gaps after every test run and every code change. And since the analysis works incrementally, it computes the update in a matter of seconds, even for very large systems.
To determine test gaps, TGA considers the chronological order of changes and test runs: New changes invalidate any previous test coverage of the changed code and open new test gaps. With subsequent testing, new coverage is recorded, which closes respective test gaps on earlier changes. Consequently, TGA can give us an update on our test gaps after every test run and every code change. And since the analysis works incrementally, it computes the update in a matter of seconds, even for very large systems.
Fig. 9 Test-gap Treemap for a UI component of Teamscale. A global analysis of remaining test gaps serves as an additional quality gate before a release a Before each Teamscale release (as of this writing, every 6 weeks) a test architect inspects all remaining test gaps on changes since the last release across the entire system. This provides us with a second quality gate, to ensure that no critical functionality accidentally slipped through testing. In this process, the architect uses the same data that was used in the development process of the features, but on a treemap that represents the entire code instead of only the code changes for an individual feature. Figure 9 shows a section of this global test-gap treemap, representing one of Teamscale’s UI components.
Fig. 9 Test-gap Treemap for a UI component of Teamscale. A global analysis of remaining test gaps serves as an additional quality gate before a release a Before each Teamscale release (as of this writing, every 6 weeks) a test architect inspects all remaining test gaps on changes since the last release across the entire system. This provides us with a second quality gate, to ensure that no critical functionality accidentally slipped through testing. In this process, the architect uses the same data that was used in the development process of the features, but on a treemap that represents the entire code instead of only the code changes for an individual feature. Figure 9 shows a section of this global test-gap treemap, representing one of Teamscale’s UI components.

Related papers

Testing run-time evolving systems
Henry Muccini

2009

Computer systems undergoing runtime evolution, such as online software-upgrades or architectural reconfigurations, must cope with changes that happen during the system execution and that might be unpredictable at design-time. The evolution requires a sequence of transition phases, where the system enters configurations emerging at runtime that could not have been validated in advance. Reasoning about such emerging behavior is difficult because previously-established system invariants do not hold, changes are implemented by both human and software agents, and externally-imposed deadlines might determine the success of the evolution. The workgroup has discussed three concrete scenarios for runtime evolution and has identified a set of challenges that are not adequately addressed by current testing approaches.

View PDFchevron_right
Avoiding bugs pro-actively by change-oriented programming
Serge Demeyer

Proceedings of the 1st Workshop on Testing Object-Oriented Systems - ETOOS '10, 2010

To minimize the cost of fixing bugs, they need to be identified as soon as possible. Testing the system is a means to detect such defects. However, when all the tests are run, it is often difficult to locate the precise error that causes a test to fail. We propose to use change-oriented programming as a means to detect bugs in a system while it is being developed. Not only may this technique assists in detecting bugs sooner, it can also be used to provide the developer with fine-grained information about the places in the program and test code that relate to the defect. We believe that this will decrease the probability of introducing bugs in a system and speed up the detection of the bugs that do get in.

View PDFchevron_right
Effective identification of failure-inducing changes
Jianjun Zhao

Proceedings of the 8th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering - PASTE '08, 2008

When regression tests fail unexpectedly after a long session of editing, it may be tedious for programmers to find out the failureinducing changes by manually inspecting all code edits. To eliminate the expensive effort spent on debugging, we present a hybrid approach, which combines both static and dynamic analysis techniques, to automatically identify the faulty changes. Our approach first uses static change impact analysis to isolate a subset of responsible changes for a failed test, then utilizes the dynamic test execution information to rank these changes according to our proposed heuristic (indicating the likelihood that they may have contributed to the failure), and finally employs an improved Three-Phase delta debugging algorithm, working from the coarse method level to the fine statement level, to find a minimal set of faulty statements.

View PDFchevron_right
Test-suite augmentation for evolving software
Raul Santelices

2008

View PDFchevron_right
Demand-driven Propagation-based Strategies for Testing Changes
Raul Santelices

2013

SUMMARY Test-suite augmentation techniques enhance test suites for software changes. In previous work, we introduced an augmentation technique that enumerates the conditions for the propagation of the effects of changes. Empirical studies showed that this technique can test changes effectively but, because of the high complexity of the technique, the experiments were small and the propagation distances from each change were limited.

View PDFchevron_right
Efficiently scripting change-resilient tests
Srinivas mangipudi

Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, 2012

In industrial practice, test cases often start out as steps described in natural language and are intended to be executed by a human. Since tests are executed repeatedly, they go through an automation process, in which they are converted to automated test scripts (or programs) that perform the test steps mechanically. Conventional test-automation techniques can be time-consuming, require specialized skills, and can produce fragile scripts. To address these limitations, we present a tool, called ATA, for automating the testautomation task. Using a novel combination of natural-language processing, backtracking exploration, and learning, ATA can significantly improve tester productivity in automating manual tests. ATA also produces change-resilient scripts, which automatically adapt themselves in the presence of certain common types of userinterface changes.

View PDFchevron_right
Efficient systematic testing for dynamically updatable software
Michael Hicks

2009

Abstract Recent years have seen significant advances in dynamic software updating (DSU) systems, which allow programs to be patched on the fly. However, a significant challenge remains: How can we ensure the act of applying a patch does not itself introduce errors? In this paper, we address this problem by presenting a new systematic testing methodology for updatable programs.

View PDFchevron_right
Static/Dynamic Test Case Generation For Software Upgrades via ARC-B and Deltatest
Pietro Braione

Validation of Evolving Software, 2015

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

View PDFchevron_right
Effective identification of failure-inducing changes: a hybrid approach
Sai Zhang

2008

Abstract When regression tests fail unexpectedly after a long session of editing, it may be tedious for programmers to find out the failure-inducing changes by manually inspecting all code edits. To eliminate the expensive effort spent on debugging, we present a hybrid approach, which combines both static and dynamic analysis techniques, to automatically identify the faulty changes.

View PDFchevron_right
Change-based test selection: an empirical evaluation
Javier Panete Pérez

Empirical Software Engineering, 2015

Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions that work well within the context of the Integrated Development Environment (IDE) are in order. Consequently, we created two variants of a test selection heuristic which exploit fine-grained changes recorded during actual development inside the IDE. One variant only considers static binding of method invocations while the other variant takes dynamic binding into account. This paper investigates the tradeoffs between these two variants in terms of the reduction (i.e., How many tests could we omit from the test suite, and how much did we gain in runtime execution?) as well as the fault detection ability of the reduced test suite (i.e., Were tests omitted erroneously?). We used our approach on three distinct cases, two open source cases-Cruisecontrol and PMD-and one industrial case-Historia. Our results show that only considering static binding reduces the test suite significantly but occasionally omits a relevant test; considering dynamic binding rarely misses a test yet often boils down to running the complete test suite. Nevertheless, our analysis provides indications on when a given variant is more appropriate.

View PDFchevron_right

References (5)

  1. Eder, S., Hauptmann, B., Junker, M., Juergens, E., Vaas, R., Prommer, K.H.: Did we test our changes? Assessing alignment between tests and development in practice. In: Proceedings of the Eighth International Workshop on Automation of Software Test (AST'13) (2013)
  2. Juergens, E., Pagano, D.: Did We Test the Right Thing? Experiences with Test Gap Analysis in Practice. Whitepaper, CQSE GmbH (2016)
  3. Juergens, E., Pagano, D., Goeb, A.: Test Impact Analysis: Detecting Errors Early Despite Large, Long-Running Test Suites. Whitepaper, CQSE GmbH (2018)
  4. Rott, J.: Empirische Untersuchung der Effektivität von Testpriorisierungsverfahren in der Praxis. Master's thesis, Technische Universität München (2019)
  5. Rott, J., Niedermayr, R., Juergens, E., Pagano, D.: Ticket coverage: putting test coverage into context. In: Proceedings of the 8th Workshop on Emerging Trends in Software Metrics (WETSoM'17) (2017)

Related papers

Change-effects analysis for effective testing and validation of evolving software
Raul Santelices

2012

The constant modification of software during its life cycle poses many challenges for developers and testers because changes might not behave as expected or may introduce erroneous side effects. For those reasons, it is of critical importance to analyze, test, and validate software every time it changes. The most common method for validating modified software is regression testing, which identifies differences in the behavior of software caused by changes and determines the correctness of those differences.

View PDFchevron_right
Applying Aggressive Propagation-based Strategies for Testing Changes
Raul Santelices

2010

View PDFchevron_right
Introducing Continuous Systematic Testing of Evolving Software
John Penix

2010

In today's evolutionary development of software, continuous testing is needed to ensure that the software is still functioning after changes. Test automation helps partly managing the large number of executions needed, but there is also a limit for how much automated tests may be executed. Then systematic approaches for test selection are needed also for automated tests. This manuscript defines this situation and outlines a general method and tool framework for its solution. Experiences from different companies are collected to illustrate how it may be set into practice.

View PDFchevron_right
Orchestration Strategies for Regression Testing of Evolving Software Systems
Renan Greca

2020

Software is an important part of modern life, and in most cases, it provides tremendous benefits to society. Unfortunately, software is highly susceptible to faults. Faults are often harmless, but even small errors can cause massive damages depending on the context. Thus, it is crucial for software developers to adopt testing techniques that can help locate faults and guarantee the function-ality of individual components. Today, more and more software is being developed in a continuously-evolving fashion, in which it is desired that changes such as new features and corrections are introduced to the end users as fast as possible. However, this desire for velocity places an additional challenge for software testing techniques , as running complete test suites can be a costly process in large-scale software. Furthermore, there is currently a disconnect between academia and industry on the subject of software testing techniques; few academically-proposed techniques are actually put into practice by software developers. We discuss the reasons and challenges behind this disconnect, and propose the development of new test orchestration strategies that aim to combine relevant software testing techniques into a complete solution that can be applied by practitioners. In order to help bring academia and industry closer together, we also propose applicability metrics that will be used to measure results of software testing techniques and compare how well each of them can perform on continuously-evolving and large-scale software. CCS CONCEPTS • Software and its engineering → Software testing and de-bugging.

View PDFchevron_right
JUnitMX - A change-aware unit testing tool
Barbara Ryder

2009 IEEE 31st International Conference on Software Engineering, 2009

Developers use unit testing to improve the quality of software systems. Current development tools for unit testing help with automating test execution, with reporting results, and with generating test stubs. However, they offer no aid for designing tests aimed specifically at exercising the effects of changes to a program. This paper describes a unit testing tool that leverages a change model to assist developers in the creation of new unit tests. The tool provides developers with quantitative feedback and detailed information about change effects, which not only facilitate the writing of more effective tests, but also motivate developers with an achievable coverage goal.

View PDFchevron_right

Related topics

  • Computer Science
  • Software Engineering
  • Software Quality
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved