Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Related Work
Experimental Planning A. Goals
B: Dataset Used to Answer Q 3
C: Remark Regarding the Datasets
Experimental Procedure
Analysis
Discussion
Conclusion
References
All Topics
Mathematics
Applied Mathematics

Know You Neighbor: Fast Static Prediction of Test Flakiness

Profile image of Breno MirandaBreno Miranda

2021, IEEE Access

https://doi.org/10.1109/ACCESS.2021.3082424
visibility

…

description

16 pages

link

1 file

Abstract

Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem. Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and k-Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 non-flaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-the-art, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion: The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.

Related papers

What is the Vocabulary of Flaky Tests? An Extended Replication
Silvia Regina Vergilio

2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC), 2021

Software systems have been continuously evolved and delivered with high quality due to the widespread adoption of automated tests. A recurring issue hurting this scenario is the presence of flaky tests, a test case that may pass or fail non-deterministically. A promising, but yet lacking more empirical evidence, approach is to collect static data of automated tests and use them to predict their flakiness. In this paper, we conducted an empirical study to assess the use of code identifiers to predict test flakiness. To do so, we first replicate most parts of the previous study of Pinto et al. (MSR 2020). This replication was extended by using a different ML Python platform (Scikit-learn) and adding different learning algorithms in the analyses. Then, we validated the performance of trained models using datasets with other flaky tests and from different projects. We successfully replicated the results of Pinto et al. (2020), with minor differences using Scikit-learn; different algorithms had performance similar to the ones used previously. Concerning the validation, we noticed that the recall of the trained models was smaller, and classifiers presented a varying range of decreases. This was observed in both intraproject and inter-projects test flakiness prediction.

View PDFchevron_right
On the use of test smells for prediction of flaky tests
Silvia Regina Vergilio

Brazilian Symposium on Systematic and Automated Software Testing

Regression testing is an important phase to deliver software with quality. However, flaky tests hamper the evaluation of test results and can increase costs. This is because a flaky test may pass or fail non-deterministically and to identify properly the flakiness of a test requires rerunning the test suite multiple times. To cope with this challenge, approaches have been proposed based on prediction models and machine learning. Existing approaches based on the use of the test case vocabulary may be context-sensitive and prone to overfitting, presenting low performance when executed in a cross-project scenario. To overcome these limitations, we investigate the use of test smells as predictors of flaky tests. We conducted an empirical study to understand if test smells have good performance as a classifier to predict the flakiness in the cross-project context, and analyzed the information gain of each test smell. We also compared the test smell-based approach with the vocabularybased one. As a result, we obtained a classifier that had a reasonable performance (Random Forest, 0.83) to predict the flakiness in the testing phase. This classifier presented better performance than vocabulary-based model for crossproject prediction. The Assertion Roulette and Sleepy Test test smell types are the ones associated with the best information gain values.

View PDFchevron_right
Change-based test selection: an empirical evaluation
Javier Panete Pérez

Empirical Software Engineering, 2015

Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions that work well within the context of the Integrated Development Environment (IDE) are in order. Consequently, we created two variants of a test selection heuristic which exploit fine-grained changes recorded during actual development inside the IDE. One variant only considers static binding of method invocations while the other variant takes dynamic binding into account. This paper investigates the tradeoffs between these two variants in terms of the reduction (i.e., How many tests could we omit from the test suite, and how much did we gain in runtime execution?) as well as the fault detection ability of the reduced test suite (i.e., Were tests omitted erroneously?). We used our approach on three distinct cases, two open source cases-Cruisecontrol and PMD-and one industrial case-Historia. Our results show that only considering static binding reduces the test suite significantly but occasionally omits a relevant test; considering dynamic binding rarely misses a test yet often boils down to running the complete test suite. Nevertheless, our analysis provides indications on when a given variant is more appropriate.

View PDFchevron_right
What is the Vocabulary of Flaky Tests?
Supun Chathuranga dissanayake

Proceedings of the 17th International Conference on Mining Software Repositories, 2020

Flaky tests are tests whose outcomes are non-deterministic. Despite the recent research activity on this topic, no effort has been made on understanding the vocabulary of flaky tests. This work proposes to automatically classify tests as flaky or not based on their vocabulary. Static classification of flaky tests is important, for example, to detect the introduction of flaky tests and to search for flaky tests after they are introduced in regression test suites. We evaluated performance of various machine learning algorithms to solve this problem. We constructed a data set of flaky and non-flaky tests by running every test case, in a set of 64k tests, 100 times (6.4 million test executions). We then used machine learning techniques on the resulting data set to predict which tests are flaky from their source code. Based on features, such as counting stemmed tokens extracted from source code identifiers, we achieved an F-measure of 0.95 for the identification of flaky tests. The best prediction performance was obtained when using Random Forest and Support Vector Machines. In terms of the code identifiers that are most strongly associated with test flakiness, we noted that job, action, and services are commonly associated with flaky tests. Overall, our results provides initial yet strong evidence that static detection of flaky tests is effective. CCS CONCEPTS • Software and its engineering → Software testing and debugging.

View PDFchevron_right
Machine Learning for Test Case Prioritization in Continuous Integration: A Comprehensive Analysis
International Journal of Advanced Networking and Applications (IJANA)

Eswar Publications, 2024

This study introduces an innovative Predictive Test Prioritization (PTP) methodology for Continuous Integration (CI), utilizing historical test case execution data. To predict the probability of success or failure in new test cases, machine learning classifiers like k-nearest Neighbors, Random Forest, Support Vector Machine (SVM), Gradient Boosting, and Logistic Regression are applied and trained on historical data. The evaluation process encompasses metrics such as F1-score, recall, accuracy, and precision, offering a nuanced understanding of the effectiveness of the classifiers. The overarching goal is to optimize test prioritization, potentially enhancing software testing efficiency. This research offers valuable insights into continuous integration systems, emphasizing the pivotal role of predictive strategies in refining testing practices and contributing to the knowledge base in CI.

View PDFchevron_right
Understanding Regression Failures through Test-Passing and Test-Failing Code Changes
Sai Zhang

Abstract—Debugging and isolating changes responsible for regression test failures are some of the most challenging aspects of modern software development. Automatic bug localization techniques reduce the manual effort developers spend examining code, for example by focusing attention on the minimal subset of recent changes that results in the test failure, or on changes to components with most dependencies or highest churn.

View PDFchevron_right
Black-Box Prediction of Flaky Test Fix Categories Using Language Models
sakina fatima

arXiv (Cornell University), 2023

Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test, causing confusion and wasting development effort. While machine learning models have been used to predict flakiness and its root causes, there is much less work on providing support to fix the problem. To address this gap, in this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis. We do this for a subset of flaky test cases where the root cause of flakiness is in the test case itself and not in the production code. Our key idea is to guide the repair process with additional knowledge about the test's flakiness in the form of its predicted fix category. Thus, we first propose a framework that automatically generates labeled datasets for 13 fix categories and trains models to predict the fix category of a flaky test by analyzing the test code only. Our experimental results using code models and few-shot learning show that we can correctly predict most of the fix categories. To show the usefulness of such fix category labels for automatically repairing flakiness, in addition to informing testers, we augment a Large Language Model (LLM) like GPT with such extra knowledge to ask the LLM for repair suggestions. The results show that our suggested fix category labels significantly enhance the capability of GPT 3.5 Turbo, in generating fixes for flaky tests.

View PDFchevron_right
A similarity-based approach for test case prioritization using historical failure data
Tanzeem Bin Noor

2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), 2015

Test case prioritization is a crucial element in software quality assurance in practice, specially, in the context of regression testing. Typically, test cases are prioritized in a way that they detect the potential faults earlier. The effectiveness of test cases, in terms of fault detection, is estimated using quality metrics, such as code coverage, size, and historical fault detection. Prior studies have shown that previously failing test cases are highly likely to fail again in the next releases, therefore, they are highly ranked, while prioritizing. However, in practice, a failing test case may not be exactly the same as a previously failed test case, but quite similar, e.g., when the new failing test is a slightly modified version of an old failing one to catch an undetected fault. In this paper, we define a class of metrics that estimate the test cases quality using their similarity to the previously failing test cases. We have conducted several experiments with five real world open source software systems, with real faults, to evaluate the effectiveness of these quality metrics. The results of our study show that our proposed similarity-based quality measure is significantly more effective for prioritizing test cases compared to existing test case quality measures.

View PDFchevron_right
Test Re-Prioritization in Continuous Testing Environments
Yuecai Zhu

2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2018

New changes are constantly and concurrently being made to large software systems. In modern continuous integration and deployment environments, each change requires a set of tests to be run. This volume of tests leads to multiple test requests being made simultaneously, which warrant prioritization of such requests. Previous work on test prioritization schedules queued tests at set time intervals. However, after a test has been scheduled it will never be reprioritized even if new higher risk tests arrive. Furthermore, as each test finishes, new information is available which could be used to reprioritize tests. In this work, we use the conditional failure probability among tests to reprioritize tests after each test run. This means that tests can be reprioritized hundreds of times as they wait to be run. Our approach is scalable because we do not depend on static analysis or coverage measures and simply prioritize tests based on their co-failure probability distributions. We named this approach CODYNAQ and in particular, we propose three prioritization variants called CODYNAQSINGLE, CODYNAQDOUBLE and CODYNAQFLEXI. We evaluate our approach on two data sets, CHROME and Google testing data. We find that our co-failure dynamic re-prioritization approach, CODYNAQ, outperforms the default order, FIFOBASELINE, finding the first failure and all failures for a change request by 31% and 62% faster, respectively. CODYNAQ also outperforms GOOGLETCP by finding the first failure 27% faster and all failures 62% faster.

View PDFchevron_right
Test-suite augmentation for evolving software
Raul Santelices

2008

View PDFchevron_right

References (53)

  1. Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, ''An empirical analysis of flaky tests,'' in Proc. 22nd ACM SIGSOFT Int. Symp. Found. Softw. Eng., New York, NY, USA, 2014, pp. 643-653.
  2. Google Testing Blog. (2008). Testing on the Toilet Avoiding Flaky Tests. Accessed: Aug. 6, 2019. [Online]. Available: https://testing.googleblog. com/2008/04/tott-avoiding-flakey-tests.html
  3. M. Fowler. (2011). Eradicating Non-Determinism in Tests. Accessed: Aug. 2, 2019. [Online]. Available: https://martinfowler.com/articles/ nonDeterminism.html
  4. P. Sudarshan. (2012). No More Flaky Tests on the Go Team. Accessed: Aug. 6, 2019. [Online]. Available: https://www.thoughtworks. com/insights/blog/no-more-flaky-tests-go-team,
  5. MDN Web Docs Mozilla. (2019). Test Verification. Accessed: Jul. 13, 2020. [Online]. Available: https://developer.mozilla.org/en-US/ docs/Mozilla/QA/Test_Verification
  6. J. Micco. (2016). Flaky Tests at Google and How Mitigate Them. Accessed: Jul. 22, 2019. [Online]. Available: https://testing.googleblog. com/2016/05/flaky-tests-at-google-and-how-we%.html
  7. C. Leong, A. Singh, M. Papadakis, Y. L. Traon, and J. Micco, ''Assessing transition-based test selection algorithms at Google,'' in Proc. 41st Int. Conf. Softw. Eng., Softw. Eng. Pract., Piscataway, NJ, USA, Mar. 2019, pp. 101-110.
  8. W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thummalapenta, ''Root causing flaky tests in a large-scale industrial setting,'' in Proc. 28th ACM SIGSOFT Int. Symp. Softw. Test. Anal., New York, NY, USA, Jul. 2019, pp. 101-111.
  9. A. Labuschagne, L. Inozemtseva, and R. Holmes, ''Measuring the cost of regression testing in practice: A study of java projects using continuous integration,'' in Proc. 11th Joint Meeting Found. Softw. Eng., New York, NY, USA, Aug. 2017, pp. 821-830.
  10. M. Beller, G. Gousios, A. Panichella, S. Proksch, S. Amann, and A. Zaidman, ''Developer testing in the IDE: Patterns, beliefs, and behav- ior,'' IEEE Trans. Softw. Eng., vol. 45, no. 3, pp. 261-284, Mar. 2017.
  11. J. Micco. (2017). The State of Continuous Integration Testing Google. Accessed: May 20, 2021. [Online]. Available: https://ai.google/ research/pubs/pub45880
  12. J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, ''DeFlaker: Automatically detecting flaky tests,'' in Proc. 40th Int. Conf. Softw. Eng., New York, NY, USA, May 2018, pp. 433-444.
  13. W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, ''IDFlakies: A framework for detecting and partially classifying flaky tests,'' in Proc. 12th IEEE Conf. Softw. Test., Validation Verification (ICST), Apr. 2019, pp. 312-322.
  14. M. Waterloo, S. Person, and S. Elbaum, ''Test analysis: Searching for faults in tests (N),'' in Proc. 30th IEEE/ACM Int. Conf. Automated Softw. Eng. (ASE), Piscataway, NJ, USA, Nov. 2015, pp. 149-154.
  15. K. Herzig and N. Nagappan, ''Empirically detecting false test alarms using association rules,'' in Proc. IEEE/ACM 37th IEEE Int. Conf. Softw. Eng., May 2015, pp. 39-48.
  16. T. M. King, D. Santiago, J. Phillips, and P. J. Clarke, ''Towards a Bayesian network model for predicting flaky automated tests,'' in Proc. IEEE Int. Conf. Softw. Qual., Rel. Secur. Companion (QRS-C), Jul. 2018, pp. 100-107.
  17. M. Harman and P. O'Hearn, ''From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis,'' in Proc. IEEE 18th Int. Work. Conf. Source Code Anal. Manipulation (SCAM), Sep. 2018, pp. 1-23.
  18. C. Ziftci and D. Cavalcanti, ''De-flake your tests: Automatically locating root causes of flaky tests in code at Google,'' in Proc. IEEE Int. Conf. Softw. Maintenance Evol. (ICSME), Sep. 2020, pp. 736-745.
  19. B. Miranda, E. Cruciani, R. Verdecchia, and A. Bertolino, ''FAST approaches to scalable similarity-based test case prioritization,'' in Proc. 40th Int. Conf. Softw. Eng., New York, NY, USA, May 2018, pp. 222-232.
  20. E. Cruciani, B. Miranda, R. Verdecchia, and A. Bertolino, ''Scalable approaches for test suite reduction,'' in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng. (ICSE), May 2019, pp. 419-429.
  21. G. Pinto, B. Miranda, S. Dissanayake, M. d'Amorim, C. Treude, and A. Bertolino, ''What is the vocabulary of flaky tests?'' in Proc. 17th Int. Conf. Mining Softw. Repositories, Jun. 2020, pp. 492-502.
  22. N. S. Altman, ''An introduction to kernel and nearest-neighbor nonpara- metric regression,'' Amer. Statist., vol. 46, no. 3, pp. 175-185, Aug. 1992.
  23. E. Kowalczyk, K. Nair, Z. Gao, L. Silberstein, T. Long, and A. Memon, ''Modeling and ranking flaky tests at Apple,'' in Proc. ACM/IEEE 42nd Int. Conf. Softw. Eng., Softw. Eng. Pract., New York, NY, USA, Jun. 2020, pp. 110-119.
  24. A. Vahabzadeh, A. M. Fard, and A. Mesbah, ''An empirical study of bugs in test code,'' in Proc. IEEE Int. Conf. Softw. Maintenance Evol. (ICSME), Washington, DC, USA, Sep. 2015, pp. 101-110.
  25. M. T. Rahman and P. C. Rigby, ''The impact of failing, flaky, and high failure tests on the number of crash reports associated with firefox builds,'' in Proc. 26th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., New York, NY, USA, Oct. 2018, pp. 857-862.
  26. S. Thorve, C. Sreshtha, and N. Meng, ''An empirical study of flaky tests in Android apps,'' in Proc. IEEE Int. Conf. Softw. Maintenance Evol. (ICSME), Washington, DC, USA, Sep. 2018, pp. 534-538.
  27. K. Presler-Marshall, E. Horton, S. Heckman, and K. Stolee, ''Wait, wait. No, tell Me. Analyzing selenium configuration effects on test flakiness,'' in Proc. IEEE/ACM 14th Int. Workshop Autom. Softw. Test (AST), Piscataway, NJ, USA, May 2019, pp. 7-13.
  28. M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, ''Understand- ing flaky tests: The developer's perspective,'' in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., Aug. 2019, pp. 830-840.
  29. W. Lam, K. Muälu, H. Sajnani, and S. Thummalapenta, ''A study on the lifecycle of flaky tests,'' in Proc. ACM/IEEE 42nd Int. Conf. Softw. Eng., New York, NY, USA, Jun. 2020, pp. 1471-1482.
  30. W. Lam, S. Winter, A. Astorga, V. Stodden, and D. Marinov, ''Understand- ing reproducibility and characteristics of flaky tests through test reruns in java projects,'' in Proc. IEEE 31st Int. Symp. Softw. Rel. Eng. (ISSRE), Oct. 2020, pp. 403-413.
  31. W. Lam, S. Winter, A. Wei, T. Xie, D. Marinov, and J. Bell, ''A large-scale longitudinal study of flaky tests,'' Proc. ACM Program. Lang., vol. 4, no. 5, pp. 1-29, Nov. 2020.
  32. A. Gyori, A. Shi, F. Hariri, and D. Marinov, ''Reliable testing: Detecting state-polluting tests to prevent test dependency,'' in Proc. Int. Symp. Softw. Test. Anal., New York, NY, USA, Jul. 2015, pp. 223-233.
  33. A. Gambi, J. Bell, and A. Zeller, ''Practical test dependency detection,'' in Proc. IEEE 11th Int. Conf. Softw. Test., Verification Validation (ICST), Apr. 2018, pp. 1-11.
  34. O. Parry, G. M. Kapfhammer, M. Hilton, and P. Mcminn, ''Flake it 'Till you make it: Using automated repair to induce and fix latent test flakiness,'' in Proc. IEEE/ACM 42nd Int. Conf. Softw. Eng. Workshops, New York, NY, USA, Jun. 2020, pp. 11-12.
  35. V. Terragni, P. Salza, and F. Ferrucci, ''A container-based infrastructure for fuzzy-driven root causing of flaky tests,'' in Proc. ACM/IEEE 42nd Int. Conf. Softw. Eng., New Ideas Emerg. Results, New York, NY, USA, Jun. 2020, pp. 69-72.
  36. A. Shi, W. Lam, R. Oei, T. Xie, and D. Marinov, ''IFixFlakies: A frame- work for automatically fixing order-dependent flaky tests,'' in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., New York, NY, USA, Aug. 2019, pp. 545-555.
  37. J. Malm, A. Causevic, B. Lisper, and S. Eldh, ''Automated analysis of flakiness-mitigating delays,'' Proc. IEEE/ACM 1st Int. Conf. Autom. Softw. Test, New York, NY, USA, Oct. 2020, pp. 81-84.
  38. A. Shi, A. Gyori, O. Legunsen, and D. Marinov, ''Detecting assumptions on deterministic implementations of non-deterministic specifications,'' in Proc. IEEE Int. Conf. Softw. Test., Verification Validation (ICST), Apr. 2016, pp. 80-90.
  39. D. Silva, L. Teixeira, and M. d'Amorim, ''Shake it! Detecting flaky tests caused by concurrency with shaker,'' in Proc. IEEE Int. Conf. Softw. Maintenance Evol. (ICSME), Sep. 2020, pp. 301-311.
  40. S. Dutta, A. Shi, R. Choudhary, Z. Zhang, A. Jain, and S. Misailovic, ''Detecting flaky tests in probabilistic and machine learning applications,'' in Proc. 29th ACM SIGSOFT Int. Symp. Softw. Test. Anal., New York, NY, USA, Jul. 2020, pp. 211-224.
  41. K. Herzig, ''Let's assume we had to pay for testing,'' Keynote at AST, 2016. [Online]. Available: https://www.slideshare.net/kim.herzig/keynote- ast-2016
  42. J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining Massive Datasets. New York, NY, USA: Cambridge Univ. Press, 2014.
  43. K. Beyer, J. Goldstein, R. Ramakrishnan, and U. Shaft, ''When is 'nearest neighbor' meaningful?'' in Proc. Int. Conf. Database Theory. Berlin, Germany: Springer, 1999, pp. 217-235.
  44. D. Achlioptas, ''Database-friendly random projections: Johnson- Lindenstrauss with binary coins,'' J. Comput. Syst. Sci., vol. 66, no. 4, pp. 671-687, Jun. 2003.
  45. P. Li, T. J. Hastie, and K. W. Church, ''Very sparse random projections,'' in Proc. 12th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2006, pp. 287-296.
  46. W. B. Johnson and J. Lindenstrauss, ''Extensions of lipschitz mappings into a Hilbert space,'' Contemp. Math., vol. 26, pp. 189-206, May 1984.
  47. A. Magen, ''Dimensionality reductions in 2 that preserve volumes and dis- tance to affine spaces,'' Discrete Comput. Geometry, vol. 38, pp. 139-153, Jul. 2007.
  48. S. Elbaum, G. Rothermel, and J. Penix, ''Techniques for improving regres- sion testing in continuous integration development environments,'' in Proc. 22nd ACM SIGSOFT Int. Symp. Found. Softw. Eng., New York, NY, USA, 2014, pp. 235-245.
  49. V. R. Basili, ''Goal question metric paradigm,'' in Encyclopedia of Soft- ware Engineering, 1994, pp. 528-532.
  50. R. Kohavi, ''A study of cross-validation and bootstrap for accuracy estima- tion and model selection,'' in Proc. 14th Int. Joint Conf. Artif. Intell., vol. 2. San Francisco, CA, USA: Morgan Kaufmann, 1995, pp. 1137-1143.
  51. J. L. Bentley, ''Multidimensional binary search trees used for associative searching,'' Commun. ACM, vol. 18, no. 9, pp. 509-517, Sep. 1975.
  52. S. M. Omohundro, Five Balltree Construction Algorithms. Berkeley, CA, USA: International Computer Science Institute Berkeley, 1989.
  53. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wessln, Experimentation in Software Engineering. Berlin, Germany: Springer, 2012. ROBERTO VERDECCHIA is currently pursuing the double Ph.D. degree in computer science with the Vrije Universiteit Amsterdam, The Nether- lands, and Gran Sasso Science Institute, L'Aquila, Italy. He is currently a Research Associate with the and Sustainability Group (S2), Vrije Uni- versiteit Amsterdam. His research interests include adoption of empirical methods to improve soft- ware development and system evolution, with par- ticular emphasis in the fields of technical debt, software architecture, software testing, and software energy efficiency. EMILIO CRUCIANI received the M.Sc. degree in engineering (computer science) from Sapienza University Rome, Rome, Italy, in 2016, and the Ph.D. degree in computer science from the Gran Sasso Science Institute, L'Aquila, Italy, in 2019. From 2019 to 2020, he was a Postdoctoral Researcher with COATI Team, INRIA Sophia Antipolis Méditerranée, France, and he is currently a Postdoctoral Researcher with the Efficient Algo- rithms Group, University of Salzburg, Austria. His research interests include the analysis of stochastic processes on com- plex networks and the design and implementation of scalable and efficient algorithms for massive datasets. BRENO MIRANDA received the master's degree in computer science from the Federal University of Pernambuco, Brazil, in 2011, and the Ph.D. degree in computer science from the University of Pisa, Italy, in 2016. He is currently an Assistant Pro- fessor with the Federal University of Pernambuco. His research interest includes software engineer- ing, with particular focus in software testing. ANTONIA BERTOLINO is currently a Research Director of the Italian National Research Coun- cil (CNR), Institute for Information Science and Technologies ''Alessandro Faedo'' (ISTI), Pisa, Italy. Her research covers a broad range of topics and techniques within software testing. She has published more than 200 papers in international journals, conferences, and workshops. She has par- ticipated to several collaborative projects, includ- ing more recently the European projects ElasTest, Learn Pad, and CHOReOS. She serves regularly in the Program Com- mittee of top conferences in software engineering, such as ESEC-FSE, ICSE, Software Testing, and ISSTA, ICST. She currently serves as a Senior Associate Editor for the Journal of Systems and Software (Elsevier), and as an Associate Editor of ACM Transactions on Software Engineering and Methodology, Empirical Software Engineering (Springer), and Journal of Software: Evolution and Process (Wiley).

Related papers

Toward static test flakiness prediction: a feasibility study
Valeria Pontillo

Proceedings of the 5th International Workshop on Machine Learning Techniques for Software Quality Evolution

Flaky tests are tests that exhibit both a passing and failing behavior when run against the same code. While researchers attempted to define approaches for detecting and addressing test flakiness, most of them suffer from scalability issues. This limitation has been recently targeted through machine learning solutions that could predict the flakiness of tests using a set of both static and dynamic metrics that would avoid the re-execution of tests. Recognizing the effort spent so far, this paper poses the first steps toward an orthogonal view of the problem, namely the classification of flaky tests using only statically computable software metrics. We propose a feasibility study on 72 projects of the iDFlakies dataset, and investigate the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells. First, we statistically assess those differences. Second, we build a logistic regression model to verify if the differences observed are still significant when the metrics are considered together. The results show a relation between test flakiness and a number of test and production code factors, indicating the possibility to build classification approaches that exploit those factors to predict test flakiness. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Empirical software validation.

View PDFchevron_right
Flakify: A Black-Box, Language Model-based Predictor for Flaky Tests
sakina fatima

ArXiv, 2021

Software testing assures that code changes do not adversely affect existing functionality. However, a test case can be flaky, i.e., passing and failing across executions, even for the same version of the source code. Flaky tests introduce overhead to software development as they can lead to unnecessary attempts to debug production or testing code. Besides rerunning test cases multiple times, which is time-consuming and computationally expensive, flaky tests can be predicted using machine learning (ML) models. However, the state-of-the-art ML-based flaky test predictors rely on pre-defined sets of features that are either project-specific, i.e., inapplicable to other projects, or require access to production code, which is not always available to software test engineers. Moreover, given the non-deterministic behavior of flaky tests, it can be challenging to determine a complete set of features that could potentially be associated with test flakiness. Therefore, in this paper, we prop...

View PDFchevron_right
Shaker: a Tool for Detecting More Flaky Tests Faster
Gauss Cordeiro

2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2021

A test case that intermittently passes or fails when performed under the same version of source code and test code is said to be flaky. The presence of flaky tests wastes testing time and effort. The most popular approach in industry to detect flakiness is ReRun. The idea behind ReRun is very simple: failing test cases are re-executed many times looking for inconsistencies in the output. Despite its simplicity, the ReRun strategy is very expensive both in terms of time and in terms of computational resources. This is particularly true for contexts where thousands of test cases are performed on a daily basis. Reducing the rerunning overhead is, thus, of utmost importance. This paper presents SHAKER, an open-source tool for detecting flakiness in time-constrained tests by adding noise in the execution environment. The main idea behind SHAKER is to add stressing tasks that compete with the test execution for the use of resources (CPU or memory). SHAKER is available as a GitHub Actions workflow that can be seamlessly integrated with any GitHub project. Alternatively, SHAKER can also be used via its provided Command Line Interface. In our evaluation, SHAKER was able to discover more flaky tests than ReRun and in a faster way (less re-executions); besides, our approach revealed tens of new flaky tests that went undetected by ReRun even after 50 re-executions. Thanks to its flexibility and ease of use, we believe that SHAKER can be useful for both practitioners and researchers.

View PDFchevron_right
Improving continuous integration with similarity-based test case selection
Azeem Ahmad

Proceedings of the 13th International Workshop on Automation of Software Test

Automated testing is an essential component of Continuous Integration (CI) and Delivery (CD), such as scheduling automated test sessions on overnight builds. That allows stakeholders to execute entire test suites and achieve exhaustive test coverage, since running all tests is often infeasible during work hours, i.e., in parallel to development activities. On the other hand, developers also need test feedback from CI servers when pushing changes, even if not all test cases are executed. In this paper we evaluate similarity-based test case selection (SBTCS) on integration-level tests executed on continuous integration pipelines of two companies. We select test cases that maximise diversity of test coverage and reduce feedback time to developers. Our results confirm existing evidence that SBTCS is a strong candidate for test optimisation, by reducing feedback time (up to 92% faster in our case studies) while achieving full test coverage using only information from test artefacts themselves.

View PDFchevron_right
An Evaluation of Machine Learning Methods for Predicting Flaky Tests
Azeem Ahmad

2020

In this paper we have investigated as a means of prevention the feasibility of using machine learning (ML) classiers for aky test prediction in project written with Python. This study compares the predictive accuracy of the three machine learning classiers (Naive Bayes, Support Vector Machines, and Random Forests) with each other. We compared our ndings with the earlier investigation of similar ML classiers for projects written in Java. Authors in this study investigated if test smells are good predictors of test akiness. As developers need to trust the predictions of ML classiers, they wish to know which types of input data or test smells cause more false negatives and false positives. We concluded that RF performed better when it comes to precision (> 90%) but provided very low recall (< 10%) as compared to NB (i.e., precision < 70% and recall >30%) and SVM (i.e., precision < 70% and recall >60%).

View PDFchevron_right

Related topics

  • Computer Science
  • Curse of Dimensionality

    • Cited by

    Toward static test flakiness prediction: a feasibility study
    VALERIA PONTILLO

    Proceedings of the 5th International Workshop on Machine Learning Techniques for Software Quality Evolution

    Flaky tests are tests that exhibit both a passing and failing behavior when run against the same code. While researchers attempted to define approaches for detecting and addressing test flakiness, most of them suffer from scalability issues. This limitation has been recently targeted through machine learning solutions that could predict the flakiness of tests using a set of both static and dynamic metrics that would avoid the re-execution of tests. Recognizing the effort spent so far, this paper poses the first steps toward an orthogonal view of the problem, namely the classification of flaky tests using only statically computable software metrics. We propose a feasibility study on 72 projects of the iDFlakies dataset, and investigate the differences between flaky and non-flaky tests in terms of 25 test and production code metrics and smells. First, we statistically assess those differences. Second, we build a logistic regression model to verify if the differences observed are still significant when the metrics are considered together. The results show a relation between test flakiness and a number of test and production code factors, indicating the possibility to build classification approaches that exploit those factors to predict test flakiness. CCS CONCEPTS • Software and its engineering → Software testing and debugging; Empirical software validation.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved