Academia.eduAcademia.edu

Outline

Title
Abstract
Figures
Introduction
Research Questions
Data Source= Within
Result
Methods
Presentation of Results
Experimental Results
Discussion
Conclusions and Future Work
References
All Topics
Computer Science
Software Engineering

Learning actionable analytics from multiple software projects

Profile image of Tim MenziesTim Menzies

2020, Empirical Software Engineering

https://doi.org/10.1007/S10664-020-09843-6
visibility

…

description

35 pages

link

1 file

Abstract

The current generation of software analytics tools are mostly prediction algorithms (e.g. support vector machines, naive bayes, logistic regression, etc). While prediction is useful, after prediction comes planning about what actions to take in order to improve quality. This research seeks methods that generate demonstrably useful guidance on "what to do" within the context of a specific software project. Specifically, we propose XTREE (for within-project planning) and BELLTREE (for cross-project planning) to generating plans that can improve software quality. Each such plan has the property that, if followed, it reduces the expected number of future defect reports. To find this expected number, planning was first applied to data from release x. Next, we looked for change in release x + 1 that conformed to our plans. This procedure was applied using a range of planners from the literature, as well as XTREE. In 10 open-source JAVA systems, several hundreds of defects were reduced in sections of the code that conformed to XTREE's plans. Further, when compared to other planners, XTREE's plans were found to be easier to implement (since they were shorter) and more effective at reducing the expected number of defects.

Figures (15)
The work in this paper is gudeied by the following research questions. Fig. 1: Relationship of this paper to our prior research.
The work in this paper is gudeied by the following research questions. Fig. 1: Relationship of this paper to our prior research.
Table 1: Sample static code attributes.
Table 1: Sample static code attributes.
Learning Actionable Analytics from Multiple Software Projects Fig. 2: The figure lists defect datasets used in this paper.
Learning Actionable Analytics from Multiple Software Projects Fig. 2: The figure lists defect datasets used in this paper.
Fig. 3: An example of how developers might use XTREE to reduce software defects.
Fig. 3: An example of how developers might use XTREE to reduce software defects.
(b) A sample of possible actions developers can take. Here a ‘+’ represents an increase, a ‘— represents a decrease, and an empty cell represents no-change. Taken from Stroggylos and Spinellis (2007); Du Bois (2006); Kataoka et al. (2002); Bryton and e Abreu (2009); Elish and Alshayeb (2011, 2012). The action highlighted in gray shows an action matching XTREE’s recommendation from Figure 3.A.
(b) A sample of possible actions developers can take. Here a ‘+’ represents an increase, a ‘— represents a decrease, and an empty cell represents no-change. Taken from Stroggylos and Spinellis (2007); Du Bois (2006); Kataoka et al. (2002); Bryton and e Abreu (2009); Elish and Alshayeb (2011, 2012). The action highlighted in gray shows an action matching XTREE’s recommendation from Figure 3.A.
Fig. 4.4: To determine which of metrics are usually changed together, we use frequent itemset mining. Our dataset is continuous in nature (see (a)) so we first discretize using Fayyad- Irani (Fayyad and Irani, 1993); this gives us a representation shown in (b). Next, we convert these into “transactions” where each file contains a list of discretized OO-metrics (see (c)). Then we use the F'P-growth algorithm to mine frequent itemsets. We return the maximal frequent itemset (as in (d)). Note: in (d) the row in |§fé@i) is the maximal frequent itemset. Fig. 4.B: To build the decision tree, we find the most informative feature,i.e., the feature which has the lowest mean entropy of splits and construct a decision tree recursively in a top-down fashion as show below.
Fig. 4.4: To determine which of metrics are usually changed together, we use frequent itemset mining. Our dataset is continuous in nature (see (a)) so we first discretize using Fayyad- Irani (Fayyad and Irani, 1993); this gives us a representation shown in (b). Next, we convert these into “transactions” where each file contains a list of discretized OO-metrics (see (c)). Then we use the F'P-growth algorithm to mine frequent itemsets. We return the maximal frequent itemset (as in (d)). Note: in (d) the row in |§fé@i) is the maximal frequent itemset. Fig. 4.B: To build the decision tree, we find the most informative feature,i.e., the feature which has the lowest mean entropy of splits and construct a decision tree recursively in a top-down fashion as show below.
Fig. 4.C: For ever test instance, we pass it down the decision tree constructed in Fig. 4.B. The node it lands is called the “start”. Next we find all the “end” nodes in the tree, i.e., those which have the lowest likelihood of defects (labeled in below). Finally, perform a random-walk to get from “start” to “end”. We use the mined itemsets from Fig. 4.A to guide the walk. When presented with multiple paths, we pick the one which has the largest overlap with the frequent items. e.g., in the below example, we would pick path (b) over path (a).
Fig. 4.C: For ever test instance, we pass it down the decision tree constructed in Fig. 4.B. The node it lands is called the “start”. Next we find all the “end” nodes in the tree, i.e., those which have the lowest likelihood of defects (labeled in below). Finally, perform a random-walk to get from “start” to “end”. We use the mined itemsets from Fig. 4.A to guide the walk. When presented with multiple paths, we pick the one which has the largest overlap with the frequent items. e.g., in the below example, we would pick path (b) over path (a).
Fig. 4: XTREE Framework
Fig. 4: XTREE Framework
Fig. 5: A simple example of computing overlap. Here a ‘-’ represents no-change. Columns shaded in gray indicate a match between developer’s changes and planner’s recommendations. Learning Actionable Analytics from Multiple Software Projects
Fig. 5: A simple example of computing overlap. Here a ‘-’ represents no-change. Columns shaded in gray indicate a match between developer’s changes and planner’s recommendations. Learning Actionable Analytics from Multiple Software Projects
Fig. 6: Sample charts to illustrate the format used to present the results.
Fig. 6: Sample charts to illustrate the format used to present the results.
Fig. 7: A count of number of test instances where the developer changes overlaps a planner recommendation. The overlaps (in the x-axis) are categorized into four ranges for every dataset (these are 0 < Overlap < 25, 26 < Overlap < 50,51 < Overlap < 75, and 76 < Overlap < 100). For each of the overlap ranges, we count the the number of instances in the validation set where overlap between the planner’s recommendation and the developers changes fell in that range. Note: Higher counts for larger overlap is better, e.g., Count([75, 100]) > Count(([0, 25)) is considered better. RQ1: How well do planners recommendations match developer actions?
Fig. 7: A count of number of test instances where the developer changes overlaps a planner recommendation. The overlaps (in the x-axis) are categorized into four ranges for every dataset (these are 0 < Overlap < 25, 26 < Overlap < 50,51 < Overlap < 75, and 76 < Overlap < 100). For each of the overlap ranges, we count the the number of instances in the validation set where overlap between the planner’s recommendation and the developers changes fell in that range. Note: Higher counts for larger overlap is better, e.g., Count([75, 100]) > Count(([0, 25)) is considered better. RQ1: How well do planners recommendations match developer actions?
Fig. 8: A count of total number defects reduced and defects increased as a result each planners recommendations. The overlaps are again categorized into four ranges for every dataset (denoted by min < Overlap < maz). For each of the overlap ranges, we count the total number of defects reduced and defects increased in the validation set for the classes that were defective in the test set as a result of overlap between the planner’s recommendation and the developers changes that fell in the given range
Fig. 8: A count of total number defects reduced and defects increased as a result each planners recommendations. The overlaps are again categorized into four ranges for every dataset (denoted by min < Overlap < maz). For each of the overlap ranges, we count the total number of defects reduced and defects increased in the validation set for the classes that were defective in the test set as a result of overlap between the planner’s recommendation and the developers changes that fell in the given range
Fig. 9: A count of total number defects reduced and defects increased as a result each planners recommendations. The overlaps are again categorized into four ranges for every dataset (denoted by min < Overlap < maz). For each of the overlap ranges, we count the total number of defects reduced and defects increased in the validation set for the classes that were defective in the test set as a result of overlap between the planner’s recommendation and the developers changes that fell in the given range Learning Actionable Analytics from Multiple Software Projects
Fig. 9: A count of total number defects reduced and defects increased as a result each planners recommendations. The overlaps are again categorized into four ranges for every dataset (denoted by min < Overlap < maz). For each of the overlap ranges, we count the total number of defects reduced and defects increased in the validation set for the classes that were defective in the test set as a result of overlap between the planner’s recommendation and the developers changes that fell in the given range Learning Actionable Analytics from Multiple Software Projects
Rahul Krishna, Tim Menzies
Rahul Krishna, Tim Menzies
(a) Defects Reduced. Higher defect reduction for larger Overlap is considered better. Fig. 10: A count of total number defects reduced and defects increased as a result each planners recommendations. The overlaps are again categorized into four ranges for every dataset (denotec by min < Overlap < maz). For each of the Overlap ranges, we count the total number o: defects reduced and defects increased in the validation set for the classes that were defective ir the test set as a result of Overlap between the planner’s recommendation and the developers changes that fell in the given range
(a) Defects Reduced. Higher defect reduction for larger Overlap is considered better. Fig. 10: A count of total number defects reduced and defects increased as a result each planners recommendations. The overlaps are again categorized into four ranges for every dataset (denotec by min < Overlap < maz). For each of the Overlap ranges, we count the total number o: defects reduced and defects increased in the validation set for the classes that were defective ir the test set as a result of Overlap between the planner’s recommendation and the developers changes that fell in the given range

Related papers

How to GENERALize Across Many Software Projects? (with case studies on Predicting Defect and Project Health)
Suvodeep Majumder

2019

Despite decades of research, SE lacks widely accepted models (that offer precise quantitative predictions) about what factors most influence software quality. This paper provides a “good news” result that such general models can be generated using a new transfer learning framework called “GENERAL”. Given a tree of recursively clustered projects (using project meta-data), GENERAL promotes a model upwards if it performs best in the lower clusters (stopping when the promoted model performs worse than the models seen at a lower level). The number of models found by GENERAL is minimal: one for defect prediction (756 projects) and less than a dozen for project health (1628 projects). Hence, via GENERAL, it is possible to make conclusions that hold across hundreds of projects at a time. Further, the models produced in this manner offer predictions that perform as well or better than prior state-of-the-art. To the best of our knowledge, this is the largest demonstration of the generalizabil...

View PDFchevron_right
From Bugs to Decision Support - Leveraging Historical Issue Reports in Software Evolution
Markus Borg

2015

Software developers in large projects work in complex information landscapes and staying on top of all relevant software artifacts is an acknowledged challenge. As software systems often evolve over many years, a large number of issue reports is typically managed during the lifetime of a system, representing the units of work needed for its improvement, e.g., defects to fix, requested features, or missing documentation. Efficient management of incoming issue reports requires the successful navigation of the information landscape of a project. In this thesis, we address two tasks involved in issue management: Issue Assignment (IA) and Change Impact Analysis (CIA). IA is the early task of allocating an issue report to a development team, and CIA is the subsequent activity of identifying how source code changes affect the existing software artifacts. While IA is fundamental in all large software projects, CIA is particularly important to safety-critical development. Our solution approach, grounded on surveys of industry practice as well as scientific literature, is to support navigation by combining information retrieval and machine learning into Recommendation Systems for Software Engineering (RSSE). While the sheer number of incoming issue reports might challenge the overview of a human developer, our techniques instead benefit from the availability of ever-growing training data. We leverage the volume of issue reports to develop accurate decision support for software evolution. We evaluate our proposals both by deploying an RSSE in two development teams, and by simulation scenarios, i.e., we assess the correctness of the RSSEs' output when replaying the historical inflow of issue reports. In total, more than 60,000 historical issue reports are involved in our studies, originating from the evolution of five proprietary systems for two companies. Our results show that RSSEs for both IA and CIA can help developers navigate large software projects, in terms of locating development teams and software artifacts. Finally, we discuss how to support the transfer of our results to industry, focusing on addressing the context dependency of our tool support by systematically tuning parameters to a specific operational setting.

View PDFchevron_right
Improving Open Source Software Process Quality Based on Defect Data Mining
Dietmar Winkler

Lecture Notes in Business Information Processing, 2012

Open Source Software (OSS) project managers often need to observe project key indicators, e.g., how much efforts are needed to finish certain tasks, to assess and improve project and product quality, e.g., by analyzing defect data from OSS project developer activities. Previous work was based on analyzing defect data of OSS projects by using correlation analysis approach for defect prediction on a combination of product and process metrics. However, this correlation analysis is focusing on the relationship between two variables without exploring the characterization of that relationship. We propose an observation framework that explores the relationship of OSS defect metrics by using data mining approach (heuristics mining algorithm). Major results show that our framework can support OSS project managers in observing project key indicators, e.g., by checking conformance between the designed and actual process models.

View PDFchevron_right
Predicting software build failure using source code metrics
Jacqui Finlay

2011

In this paper, we describe the extraction of source code metrics from the Jazz repository and the application of data mining techniques to identify the most useful of those metrics for predicting the success or failure of an attempt to construct a working instance of the software product. We present results from a study using the J48 classification method used in conjunction with a number of attribute selection strategies applied to a set of source code metrics calculated from the code base at the beginning of a build cycle. The results indicate that only a relatively small number of the available software metrics that we considered have any significance for predicting the outcome of a build. These significant metrics are discussed and implication of the results discussed, particularly the relative difficulty of being able to predict failed build attempts. The results also indicate that there is some scope for predicting the outcomes of an attempt to construct a working instance of the software product by analysing the characteristics of the source code to be changed. This provides the opportunity for software project managers to estimate the risk exposure of the planned changes in the build prior to commencing the coding activities.

View PDFchevron_right
Learning GENERAL Principles from Hundreds of Software Projects
Suvodeep Majumder

2019

Managers and practitioners become dubious about software analytics when its conclusions keep changing as we look at new projects. GENERAL is a new approach for quickly finding conclusions that generalize across hundreds of projects. This algorithm (a) removes spurious attributes via feature selection; (b) fixes training data imbalance via synthetic instances; (c) recursively clusters the project data; (d) finds the best model within any cluster, then promotes it up the cluster tree; (e) returns the model promoted to the top. GENERAL is much faster than prior methods (4.8 hours versus 204 hours our case studies) and theoretically scales better (O(N^2/m) versus O(N^2), which is a large reduction since often we find m&gt;20 clusters). When tested on 756 Github projects, a single defect prediction model generalized over all those projects while also being useful and insightful and generalizable; i.e. that model worked just as well as 756 separate models learned from each project; and th...

View PDFchevron_right
Performance of Defect Prediction in Rapidly Evolving Software
Roberto Pietrantuono

Defect prediction techniques allow spotting modules (or commits) likely to contain (introduce) a defect by training models with product or process metrics – thus supporting testing, code integration, and release decisions. When applied to processes where software changes rapidly, conventional techniques might fail, as trained models are not thought to evolve along with the software. In this study, we analyze the performance of defect prediction in rapidly evolving software. Framed in a high commit frequency context, we set up an approach to continuously refine prediction models by using new commit data, and predict whether or not an attempted commit is going to introduce a bug. An experiment is set up on the Eclipse JDT software to assess the prediction ability trend. Results enable to leverage defect prediction potentials in modern development paradigms with short release cycle and high code variability.

View PDFchevron_right
Project data incorporating qualitative factors for improved software defect prediction
Norman Fenton

Proceedings - ICSE 2007 Workshops: Third International Workshop on Predictor Models in Software Engineering, PROMISE'07, 2007

To make accurate predictions of attributes like defects found in complex software projects we need a rich set of process factors. We have developed a causal model that includes such process factors, both quantitative and qualitative. The factors in the model were identified as part of a major collaborative project. A challenge for such a model is getting the data needed to validate it. We present a dataset, elicited from 31 completed software projects in the consumer electronics industry, which we used for validation. The data were gathered using a questionnaire distributed to managers of recent projects. The dataset will be of interest to other researchers evaluating models with similar aims. We make both the dataset and causal model available for research use.

View PDFchevron_right
Topic-based defect prediction
thanh phuong

Proceeding of the 33rd international conference on Software engineering - ICSE '11, 2011

Defects are unavoidable in software development and fixing them is costly and resource-intensive. To build defect prediction models, researchers have investigated a number of factors related to the defect-proneness of source code, such as code complexity, change complexity, or socio-technical factors. In this paper, we propose a new approach that emphasizes on technical concerns/functionality of a system. In our approach, a software system is viewed as a collection of software artifacts that describe different technical concerns/aspects. Those concerns are assumed to have different levels of defect-proneness, thus, cause different levels of defectproneness to the relevant software artifacts. We use topic modeling to measure the concerns in source code, and use them as the input for machine learning-based defect prediction models. Preliminary result on Eclipse JDT shows that the topic-based metrics have high correlation to the number of bugs (defect-proneness), and our topic-based defect prediction has better predictive performance than existing state-of-the-art approaches.

View PDFchevron_right
Predicting Bugs in Distributed Large Scale Software Systems Development
Rathinaswamy Govindan
View PDFchevron_right
Integrated Approach to Software Defect Prediction
EBUBEOGU AMARACHUKWU FELIX STUDENT

IEEE Access

Software defect prediction provides actionable outputs to software teams while contributing to industrial success. Empirical studies have been conducted on software defect prediction for both crossproject and within-project defect prediction. However, existing studies have yet to demonstrate a method of predicting the number of defects in an upcoming product release. This paper presents such a method using predictor variables derived from the defect acceleration, namely, the defect density, defect velocity, and defect introduction time, and determines the correlation of each predictor variable with the number of defects. We report the application of an integrated machine learning approach based on regression models constructed from these predictor variables. An experiment was conducted on ten different data sets collected from the PROMISE repository, containing 22 838 instances. The regression model constructed as a function of the average defect velocity achieved an adjusted R-square of 98.6%, with a p-value of < 0.001. The average defect velocity is strongly positively correlated with the number of defects, with a correlation coefficient of 0.98. Thus, it is demonstrated that this technique can provide a blueprint for program testing to enhance the effectiveness of software development activities. INDEX TERMS Software defect prediction, machine learning, number of defects, defect velocity, class imbalance. EBUBEOGU AMARACHUKWU FELIX (GS'17) is currently pursuing the Ph.D. degree with the Department of Software Engineering, University of Malaya, Malaysia. His research interests include machine learning, software quality, software maintenance, and big data. He is a Student Member of ACM. SAI PECK LEE (M'10) received the Ph.D. degree in computer science from Université Paris 1 Panthéon-Sorbonne. She is currently a Professor with the Department of Software Engineering, University of Malaya. She has authored or co-authored an academic book, book chapters, and over 100 papers in various refereed journals and conference proceedings. Her current research interests include requirements engineering, software traceability and clustering, software quality, software reuse, object-oriented techniques, and CASE tools development. She has also actively participated in conference program committees and is currently in several experts review panels, both locally and internationally.

View PDFchevron_right

References (104)

  1. Al Dallal J, Briand LC (2010) An object-oriented high-level design-based class cohesion metric. Information and software technology 52(12):1346-1361
  2. Altman E (1999) Constrained Markov decision processes, vol 7. CRC Press Alves TL, Ypma C, Visser J (2010) Deriving metric thresholds from benchmark data. In: 2010 IEEE Int. Conf. Softw. Maint., IEEE, pp 1-10, DOI 10.1109/ICSM.2010. 5609747
  3. Andrews JH, Li FCH, Menzies T (2007) Nighthawk: A Two-Level Genetic-Random Unit Test Data Generator. In: IEEE ASE'07
  4. Andrews JH, Menzies T, Li FCH (2010) Genetic Algorithms for Randomized Unit Testing. IEEE Transactions on Software Engineering
  5. Baier SSJA, McIlraith SA (2009) Htn planning with preferences. In: 21st Int. Joint Conf. on Artificial Intelligence, pp 1790-1797
  6. Bansiya J, Davis CG (2002) A hierarchical model for object-oriented design quality assessment. IEEE Trans Softw Eng 28(1):4-17, DOI 10.1109/32.979986
  7. Begel A, Zimmermann T (2014) Analyze this! 145 questions for data scientists in software engineering. In: Proc. 36th Intl. Conf. Software Engineering (ICSE 2014), ACM Bellman R (1957) A markovian decision process. Indiana Univ Math J 6:679-684
  8. Bender R (1999) Quantitative risk assessment in epidemiological studies investigating threshold effects. Biometrical Journal 41(3):305-319
  9. Bener A, Misirli AT, Caglayan B, Kocaguneli E, Calikli G (2015) Lessons learned from software analytics in practice. In: The Art and Science of Analyzing Software Data, Elsevier, pp 453-489
  10. Blue D, Segall I, Tzoref-Brill R, Zlotnick A (2013) Interaction-based test-suite min- imization. In: Proc. the 2013 Intl. Conf. Software Engineering, IEEE Press, pp 182-191
  11. Boehm B (1981) Software Engineering Economics. Prentice Hall Boehm B, Horowitz E, Madachy R, Reifer D, Clark BK, Steece B, Brown AW, Chulani S, Abts C (2000) Software Cost Estimation with Cocomo II. Prentice Hall Bryton S, e Abreu FB (2009) Strengthening refactoring: towards software evolution with quantitative and experimental grounds. In: Software Engineering Advances, 2009. ICSEA'09. Fourth Intl. Conf., IEEE, pp 570-575
  12. Cheng B, Jensen A (2010) On the use of genetic programming for automated refactor- ing and the introduction of design patterns. In: Proc. 12th Annual Conf. Genetic and Evolutionary Computation, ACM, New York, NY, USA, GECCO '10, pp 1341-1348, DOI 10.1145/1830483.1830731
  13. Chidamber SR, Kemerer CF (1994) A metrics suite for object oriented design. IEEE Transactions on software engineering 20(6):476-493
  14. Chidamber SR, Darcy DP, Kemerer CF (1998) Managerial use of metrics for object- oriented software: An exploratory analysis. IEEE Transactions on software Engi- neering 24(8):629-639
  15. Cortes C, Vapnik V (1995) Support-vector networks. Machine learning 20(3):273-297
  16. Cui X, Potok T, Palathingal P (2005) Document clustering using particle swarm optimization. . . . Intelligence Symposium, 2005 . . .
  17. Czerwonka J, Das R, Nagappan N, Tarvo A, Teterev A (2011) Crane: Failure prediction, change analysis and test prioritization in practice -experiences from windows. In: Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth Intl. Conference on, pp 357 -366
  18. Deb K, Jain H (2014) An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints. Evolutionary Computation, IEEE Transactions on 18(4):577-601, DOI 10.1109/TEVC.2013.2281535
  19. Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A Fast Elitist Multi-Objective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6:182-197
  20. Devanbu P, Zimmermann T, Bird C (2016) Belief & evidence in empirical software engineering. In: Proc. 38th Intl. Conf. Software Engineering, ACM, pp 108-119
  21. Du Bois B (2006) A study of quality improvements by refactoring Elish K, Alshayeb M (2011) A classification of refactoring methods based on software quality attributes. Arabian Journal for Science and Engineering 36(7):1253-1267
  22. Elish K, Alshayeb M (2012) Using software quality attributes to classify refactoring to patterns. JSW 7(2):408-419
  23. Fayyad U, Irani K (1993) Multi-interval discretization of continuous-valued attributes for classification learning. NASA JPL Archives
  24. Fu W, Menzies T, Shen X (2016) Tuning for software analytics: is it really necessary? Information and Software Technology (submitted)
  25. Ghallab M, Nau D, Traverso P (2004) Automated Planning: theory and practice. Elsevier
  26. Ghotra B, McIntosh S, Hassan AE (2015) Revisiting the impact of classification techniques on the performance of defect prediction models. In: 37th ICSE-Volume 1, IEEE Press, pp 789-800
  27. Graves TL, Karr AF, Marron JS, Siy H (2000) Predicting fault incidence using software change history. IEEE Transactions on software engineering 26(7):653-661
  28. Guo X, Hernández-Lerma O (2009) Continuous-time markov decision processes. Continuous-Time Markov Decision Processes pp 9-18
  29. Halstead MH (1977) Elements of software science, vol 7. Elsevier New York Han J, Cheng H, Xin D, Yan X (2007) Frequent pattern mining: current status and future directions. Data mining and knowledge discovery 15(1):55-86
  30. Harman M, Mansouri SA, Zhang Y (2009) Search based software engineering: A com- prehensive analysis and review of trends techniques and applications. Department of Computer Science, KingâĂŹs College London, Tech Rep TR-09-03
  31. Harman M, McMinn P, De Souza J, Yoo S (2011) Search based software engineering: Techniques, taxonomy, tutorial. Search 2012:1-59, DOI 10.1007/978-3-642-25231-0\
  32. Henard C, Papadakis M, Harman M, Traon YL (2015) Combining multi-objective search and constraint solving for configuring large software product lines. In: 2015 IEEE/ACM 37th IEEE Intl. Conf. Software Engineering, vol 1, pp 517-528, DOI 10.1109/ICSE.2015.69
  33. Hihn J, Menzies T (2015) Data mining methods and cost estimation models: Why is it so hard to infuse new ideas? In: 2015 30th IEEE/ACM Intl. Conf. Automated Software Engineering Workshop (ASEW), pp 5-9, DOI 10.1109/ASEW.2015.27
  34. Ii PG, Menzies T, Williams S, El-Rawas O (2009) Understanding the value of software engineering technologies. In: Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, IEEE Computer Society, pp 52-61
  35. Jing X, Wu G, Dong X, Qi F, Xu B (2015) Heterogeneous cross-company defect prediction by unified metric representation and cca-based transfer learning. In: FSE'15
  36. Jørgensen M, Gruschke TM (2009) The impact of lessons-learned sessions on effort estimation and uncertainty assessments. Software Engineering, IEEE Transactions on 35(3):368 -383
  37. Jureczko M, Madeyski L (2010) Towards identifying software project clusters with regard to defect prediction. In: Proc. 6th Int. Conf. Predict. Model. Softw. Eng. - PROMISE '10, ACM Press, New York, New York, USA, p 1, DOI 10.1145/1868328. 1868342
  38. Kataoka Y, Imai T, Andou H, Fukaya T (2002) A quantitative evaluation of maintain- ability enhancement by refactoring. In: Software Maintenance, 2002. Proceedings. Intl. Conf., IEEE, pp 576-585
  39. Kocaguneli E, Menzies T (2011) How to find relevant data for effort estimation? In: Empirical Software Engineering and Measurement (ESEM), 2011 Intl. Symposium on, IEEE, pp 255-264
  40. Kocaguneli E, Menzies T, Bener A, Keung J (2012) Exploiting the essential as- sumptions of analogy-based effort estimation. IEEE Transactions on Software Engineering 28:425-438
  41. Kocaguneli E, Menzies T, Mendes E (2015) Transfer learning in effort estimation. Empirical Software Engineering 20(3):813-843, DOI 10.1007/s10664-014-9300-5
  42. Krall J, Menzies T, Davies M (2015) Gale: Geometric active learning for search-based software engineering. IEEE Transactions on Software Engineering 41(10):1001-1018
  43. Krishna R (2017) Learning effective changes for software projects. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, IEEE Press, pp 1002-1005
  44. Krishna R, Menzies T (2015) Actionable = cluster + contrast? In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering Work- shop (ASEW), pp 14-17, DOI 10.1109/ASEW.2015.23
  45. Krishna R, Menzies T (2018) Bellwethers: A baseline method for transfer learn- ing. IEEE Transactions on Software Engineering pp 1-1, DOI 10.1109/TSE.2018. 2821670
  46. Krishna R, Menzies T, Fu W (2016) Too much automation? the bellwether effect and its implications for transfer learning. In: Proc. 31st IEEE/ACM Intl. Conf. Automated Software Engineering -ASE 2016, ACM Press, New York, New York, USA, pp 122-131, DOI 10.1145/2970276.2970339
  47. Krishna R, Menzies T, Layman L (2017a) Less is more: Minimizing code reorganization using XTREE. Information and Software Technology DOI 10.1016/j.infsof.2017.03. 012, 1609.03614
  48. Krishna R, Menzies T, Layman L (2017b) Less is more: Minimizing code reorganization using xtree. Information and Software Technology 88:53-66
  49. Le Goues C, Dewey-Vogt M, Forrest S, Weimer W (2012) A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In: 2012 34th
  50. Intl. Conf. Software Engineering (ICSE), IEEE, pp 3-13, DOI 10.1109/ICSE.2012. 6227211
  51. Le Goues C, Holtschulte N, Smith EK, Brun Y, Devanbu P, Forrest S, Weimer W (2015) The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. IEEE Transactions on Software Engineering 41(12):1236-1256, DOI 10.1109/TSE.2015.2454513
  52. Lemon B, Riesbeck A, Menzies T, Price J, D'Alessandro J, Carlsson R, Prifiti T, Peters F, Lu H, Port D (2009) Applications of Simulation and AI Search: Assessing the Relative Merits of Agile vs Traditional Software Development. In: IEEE ASE'09
  53. Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings. IEEE Trans Softw Eng 34(4):485-496, DOI 10.1109/TSE.2008.35
  54. Lewis C, Lin Z, Sadowski C, Zhu X, Ou R, Whitehead Jr EJ (2013) Does bug prediction support human developers? findings from a google case study. In: Pro- ceedings of the 2013 International Conference on Software Engineering, IEEE Press, Piscataway, NJ, USA, ICSE '13, pp 372-381, URL http://dl.acm.org/citation. cfm?id=2486788.2486838
  55. Lowry M, Boyd M, Kulkami D (1998) Towards a theory for integration of mathemat- ical verification and empirical testing. In: Automated Software Engineering, 1998. Proceedings. 13th IEEE International Conference on, IEEE, pp 322-331
  56. Madeyski L, Jureczko M (2015) Which process metrics can significantly improve defect prediction models? an empirical study. Software Quality Journal 23(3):393-422
  57. McCabe TJ (1976) A complexity measure. IEEE Transactions on software Engineering (4):308-320
  58. Mensah S, Keung J, MacDonell SG, Bosu MF, Bennin KE (2018) Investigating the significance of the bellwether effect to improve software effort prediction: Further empirical study. IEEE Transactions on Reliability (99):1-23
  59. Menzies T, Raffo D, on Setamanit S, Hu Y, Tootoonian S (2002) Model-based tests of truisms. In: Proceedings of IEEE ASE 2002, available from http://menzies. us/pdf/02truisms.pdf
  60. Menzies T, Dekhtyar A, Distefano J, Greenwald J (2007a) Problems with Precision: A Response to "Comments on 'Data Mining Static Code Attributes to Learn Defect Predictors'". IEEE Transactions on Software Engineering 33(9):637-640, DOI 10.1109/TSE.2007.70721
  61. Menzies T, Elrawas O, Hihn J, Feather M, Madachy R, Boehm B (2007b) The business case for automated software engineering. In: Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, ACM, pp 303-312
  62. Menzies T, Greenwald J, Frank A (2007c) Data mining static code attributes to learn defect predictors. IEEE transactions on software engineering 33(1):2-13
  63. Menzies T, Greenwald J, Frank A (2007d) Data mining static code attributes to learn defect predictors. IEEE Transactions on Software Engineering Available from http://menzies.us/pdf/06learnPredict.pdf
  64. Menzies T, Williams S, Boehm B, Hihn J (2009) How to avoid drastic software process change (using stochastic stability). In: Proceedings of the 31st International Conference on Software Engineering, IEEE Computer Society, Washington, DC, USA, ICSE '09, pp 540-550, DOI 10.1109/ICSE.2009.5070552, URL http://dx. doi.org/10.1109/ICSE.2009.5070552
  65. Menzies T, Butcher A, Cok D, Marcus A, Layman L, Shull F, Turhan B, Zimmermann T (2013) Local versus Global Lessons for Defect Prediction and Effort Estimation. IEEE Transactions on Software Engineering 39(6):822-834, DOI 10.1109/TSE. 2012.83
  66. Menzies T, Krishna R, Pryor D (2016) The promise repository of empirical software engineering data. north carolina state university, department of computer science
  67. Metzger A, Pohl K (2014) Software product line engineering and variability manage- ment: achievements and challenges. In: Proc. on Future of Software Engineering, ACM, pp 70-84
  68. Mkaouer MW, Kessentini M, Bechikh S, Deb K, Ó Cinnéide M (2014) Recommenda- tion system for software refactoring using innovization and interactive dynamic op- timization. In: Proc. 29th ACM/IEEE Intl. Conf. Automated Software Engineering, ACM, New York, NY, USA, ASE '14, pp 331-336, DOI 10.1145/2642937.2642965
  69. Moghadam IH (2011) Search Based Software Engineering: Third Intl. Symposium, SSBSE 2011, Szeged, Hungary, September 10-12, 2011. Proceedings, Springer Berlin Heidelberg, Berlin, Heidelberg, chap Multi-level Automated Refactoring Using Design Exploration, pp 70-75. DOI 10.1007/978-3-642-23716-4 9
  70. Nagappan N, Ball T (2005) Static analysis tools as early indicators of pre-release defect density. In: Proc. 27th Intl. Conf. Software engineering, ACM, pp 580-586
  71. Nam J, Kim S (2015a) Heterogeneous defect prediction. In: Proc. 2015 10th Jt. Meet. Found. Softw. Eng. -ESEC/FSE 2015, ACM Press, New York, New York, USA, pp 508-519, DOI 10.1145/2786805.2786814
  72. Nam J, Kim S (2015b) Heterogeneous defect prediction. In: Proc. 2015 10th Jt. Meet. Found. Softw. Eng. -ESEC/FSE 2015, ACM Press, New York, New York, USA, pp 508-519, DOI 10.1145/2786805.2786814
  73. Nam J, Pan SJ, Kim S (2013a) Transfer defect learning. In: Proc. Intl. Conf. on Software Engineering, pp 382-391, DOI 10.1109/ICSE.2013.6606584
  74. Nam J, Pan SJ, Kim S (2013b) Transfer defect learning. In: Proc. Intl. Conf. Software Engineering, pp 382-391, DOI 10.1109/ICSE.2013.6606584
  75. Nam J, Fu W, Kim S, Menzies T, Tan L (2017) Heterogeneous defect prediction. IEEE Transactions on Software Engineering PP(99):1-1, DOI 10.1109/TSE.2017.2720603
  76. Nayrolles M, Hamou-Lhadj A (2018) Clever: Combining code metrics with clone detection for just-in-time fault prevention and resolution in large industrial projects. In: Mining Software Repositories O'Keeffe M, Cinnéide MO (2008) Search-based refactoring: An empirical study. J Softw Maint Evol 20(5):345-364, DOI 10.1002/smr.v20:5
  77. O'Keeffe MK, Cinneide MO (2007) Getting the most from search-based refactoring. In: Proc. 9th Annual Conf. Genetic and Evolutionary Computation, ACM, New York, NY, USA, GECCO '07, pp 1114-1120, DOI 10.1145/1276958.1277177
  78. Oliveira P, Valente MT, Lima FP (2014) Extracting relative thresholds for source code metrics. In: 2014 Software Evolution Week -IEEE Conf. Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), IEEE, pp 254-263, DOI 10.1109/CSMR-WCRE.2014.6747177
  79. Ostrand TJ, Weyuker EJ, Bell RM (2004) Where the bugs are. In: ISSTA '04: Proc. the 2004 ACM SIGSOFT Intl. symposium on Software testing and analysis, ACM, New York, NY, USA, pp 86-96
  80. Passos C, Braun AP, Cruzes DS, Mendonca M (2011) Analyzing the impact of beliefs in software project practices. In: ESEM'11
  81. Peters F, Menzies T, Layman L (2015) LACE2: Better privacy-preserving data sharing for cross project defect prediction. In: Proc. Intl. Conf. Software Engineering, vol 1, pp 801-811, DOI 10.1109/ICSE.2015.92
  82. Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: 2013 35th International Conference on Software Engineering (ICSE), IEEE, pp 432-441
  83. Rathore SS, Kumar S (2019) A study on software fault prediction techniques. Artificial Intelligence Review 51(2):255-327, DOI 10.1007/s10462-017-9563-5, URL https: //doi.org/10.1007/s10462-017-9563-5
  84. Ruhe G (2010) Product release planning: methods, tools and applications. CRC Press Ruhe G, Greer D (2003) Quantitative studies in software release planning under risk and resource constraints. In: Empirical Software Engineering, 2003. ISESE 2003. Proceedings. 2003 Intl. Symposium on, IEEE, pp 262-270
  85. Russell S, Norvig P (1995) Artificial Intelligence: A Modern Approach. Prentice-Hall, Egnlewood Cliffs
  86. Sayyad AS, Ingram J, Menzies T, Ammar H (2013) Scalable product line configuration: A straw to break the camel's back. In: Automated Software Engineering (ASE), 2013 IEEE/ACM 28th Intl. Conf., IEEE, pp 465-474
  87. Sharma A, Jiang J, Bommannavar P, Larson B, Lin J (2016) Graphjet: real-time con- tent recommendations at twitter. Proceedings of the VLDB Endowment 9(13):1281- 1292
  88. Shatnawi R (2010) A quantitative investigation of the acceptable risk levels of object-oriented metrics in open-source systems. IEEE Transactions on Software Engineering 36(2):216-225, DOI 10.1109/TSE.2010.9
  89. Shatnawi R, Li W (2008) The effectiveness of software metrics in identifying error- prone classes in post-release software evolution process. Journal of systems and software 81(11):1868-1882
  90. Shull F, ad B Boehm VB, Brown A, Costa P, Lindvall M, Port D, Rus I, Tesoriero R, Zelkowitz M (2002) What we have learned about fighting defects. In: Proceedings of 8th International Software Metrics Symposium, Ottawa, Canada, pp 249-258
  91. Son TC, Pontelli E (2006) Planning with preferences using logic programming. Theory and Practice of Logic Programming 6(5):559-607
  92. Stroggylos K, Spinellis D (2007) Refactoring-does it improve software quality? In: Software Quality, 2007. WoSQ'07: ICSE Workshops 2007. Fifth Intl. Workshop on, IEEE, pp 10-10
  93. Tallam S, Gupta N (2006) A concept analysis inspired greedy algorithm for test suite minimization. ACM SIGSOFT Software Engineering Notes 31(1):35-42
  94. Theisen C, Herzig K, Morrison P, Murphy B, Williams L (2015) Approximating attack surfaces with stack traces. In: ICSE'15
  95. Turhan B, Menzies T, Bener AB, Di Stefano J (2009) On the relative value of cross-company and within-company data for defect prediction. Empirical Software Engineering 14(5):540-578
  96. Turhan B, Tosun A, Bener A (2011) Empirical evaluation of mixed-project defect prediction models. In: Software Engineering and Advanced Applications (SEAA), 2011 37th EUROMICRO Conf., IEEE, pp 396-403
  97. Weimer W, Nguyen T, Le Goues C, Forrest S (2009) Automatically finding patches using genetic programming. In: Proc. Intl. Conf. Software Engineering, IEEE, pp 364-374, DOI 10.1109/ICSE.2009.5070536
  98. Wooldridge M, Jennings NR (1995) Intelligent agents: Theory and practice. The knowledge engineering review 10(2):115-152
  99. Ying R, He R, Chen K, Eksombatchai P, Hamilton WL, Leskovec J (2018) Graph convolutional neural networks for web-scale recommender systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, pp 974-983
  100. Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22(2):67-120
  101. Zhang Q, Li H (2007) MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition. Evolutionary Computation, IEEE Transactions on 11(6):712-731, DOI 10.1109/TEVC.2007.892759
  102. Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction. Proc 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering on European software engineering conference and foundations of software engineering symposium -E p 91, DOI 10.1145/1595696.1595713
  103. Zitzler E, Künzli S (2004) Indicator-Based Selection in Multiobjective Search. In: Parallel Problem Solving from Nature -PPSN VIII, Lecture Notes in Computer Science, vol 3242, Springer Berlin Heidelberg, pp 832-842, DOI 10.1007/978-3-540-30217-9\ 84
  104. Zitzler E, Laumanns M, Thiele L (2002) SPEA2: Improving the Strength Pareto Evolutionary Algorithm for Multiobjective Optimization. In: Evolutionary Methods for Design, Optimisation, and Control, CIMNE, Barcelona, Spain, pp 95-100

Related papers

From Prediction to Planning: Improving Software Quality with BELLTREE
Tim Menzies

2018

The current generation of software analytics tools are mostly prediction algorithms (e.g. support vector machines, naive bayes, logistic regression, etc). While prediction is useful, after prediction comes planning about what actions to take in order to improve quality. This research seeks methods that generate demonstrably useful guidance on &#39;&#39;what to do&#39;&#39; within the context of a specific software project. Specifically, we propose XTREE (for within-project planning) and BELLTREE (for cross-project planning) to generating plans that can improve software quality. Each such plan has the property that, if followed, it reduces the probability of future defect reports. When compared to other planning algorithms from the SE literature, we find that this new approach is most effective at learning plans from one project, then applying those plans to another. In 10 open-source JAVA systems, several hundreds of defects were reduced in sections of the code that followed the pla...

View PDFchevron_right
Improved software defect prediction
Norman Fenton

2005

Although a number of approaches have been taken to quality prediction for software, none have achieved widespread applicability. This paper describes a single model to combine the diverse forms of, often causal, evidence available in software development in a more natural and efficient way than done previously. We use Bayesian Networks as the appropriate formalism for representing defect introduction, detection and removal processes throughout any life-cycle. The approach combines subjective judgements from experienced project managers and available defect rate data to produce a risk map and use this to forecast and control defect rates. Moreover, the risk map more naturally mirrors real world influences without any distracting mathematical formality. The paper focuses on the extensive validation of the approach within Philips Consumer Electronics (dozens of diverse projects across Philips internationally). The resulting model (packaged within a commercial software tool, AgenaRisk, usable by project managers) is now being used to predict defect rates at various testing and operational phases. The results of the validation confirm that the approach is scalable, robust and more accurate that can be achieved using classical methods. We have found 95% correlation between actual and predicted defects. The defect prediction models incorporate cutting-edge ideas and results from software metrics and process improvement research and package them as risk templates that can either be applied either offthe-shelf or after calibrating them to local conditions and to suit the software development processes in use.

View PDFchevron_right
Mining Data from Multiple Software Development Projects
Naeem Seliya

2009 IEEE International Conference on Data Mining Workshops, 2009

A large system often goes through multiple software project development cycles, in part due to changes in operation and development environments. For example, rapid turnover of the development team between releases can influence software quality, making it important to mine software project data over multiple system releases when building defect predictors. Data collection of software attributes are often conducted independent of the quality improvement goals, leading to the availability of a large number of attributes for analysis. Given the problems associated with variations in development process, data collection, and quality goals from one release to another emphasizes the importance of selecting a best-set of software attributes for software quality prediction. Moreover, it is intuitive to remove attributes that do not add to, or have an adverse effect on, the knowledge of the consequent model. Based on real-world software projects' data, we present a large case study that compares wrapper-based feature ranking techniques (WRT) and our proposed hybrid feature selection (HFS) technique. The comparison is done using both threefold cross-validation (CV) and threefold cross-validation with risk impact (CVR). It is shown that HFS is better than WRT, while CV is superior to CVR.

View PDFchevron_right
DEFECT / EFFORT PREDICTION MODELS IN SOFTWARE INTERNATIONAL JOURNAL OF MANAGEMENT (IJM) IJM © I A E M E
iaeme iaeme

Planning software maintenance work is a key factor for a successful maintenance project and for better project scheduling, monitoring, and control. To this aim, effort estimation is a valuable asset to maintenance managers in planning maintenance activities and performing cost/benefits analysis. Software practitioners recognize the importance of realistic estimates of effort to the successful management of software projects. Having realistic estimates at an early stage in a project's life cycle allow project managers and software organizations to manage resources effectively. Prediction is a necessary part of an effective process, be it authoring, design, testing, or Web development as a whole. Maintenance projects may range from ordinary projects requiring simple activities of understanding, impact analysis and modifications, to extraordinary projects requiring complex interventions such as encapsulation, reuse, reengineering, migration, and retirement. Moreover, software costs are the result of a large number of parameters so any estimation or control technique must reflect a large number of complex and dynamic factors. This paper mainly covers the models related to Fault predictions well ahead of the execution so that it gives an insight to the managers to take appropriate decisions. Data about the defects found during software testing is recorded in software defect reports or bug reports. The data consists of defect information including defect number at various testing stages, complexity (of the defect), severity, information about the component to which the defect belongs, tester, and person fixing the defect. Reliability models mainly use data about the number of defects and its corresponding International Journal of Management (IJM) 131 time to predict the remaining number of defects. This research work proposes an empirical approach to systematically elucidate useful information from software defect reports by (1) employing a data exploration technique that analyzes relationships between software quality of different releases using appropriate statistics, and (2) constructing predictive models for forecasting time for fixing defects using existing machine learning and data mining techniques.

View PDFchevron_right
Predicting Re-opened Bugs: A Case Study on the Eclipse Project
ahmed Hassan

2010 17th Working Conference on Reverse Engineering, 2010

Bug fixing accounts for a large amount of the software maintenance resources. Generally, bugs are reported, fixed, verified and closed. However, in some cases bugs have to be re-opened. Re-opened bugs increase maintenance costs, degrade the overall user-perceived quality of the software and lead to unnecessary rework by busy practitioners.

View PDFchevron_right

Related topics

  • Empirical Software Engineering
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved