Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Background
Code Smells Data
Issue Lifetime Data
Effort Estimation Dataset
Defect Datasets
Methodology
Learning Methods
Statistics
Results
Understanding Data Visualization
Result 2
Result 4
Discussion
Conclusion
References
All Topics
Computer Science
Information Systems

Bellwethers: A Baseline Method For Transfer Learning

Profile image of Tim MenziesTim Menzies

2018, IEEE Transactions on Software Engineering

https://doi.org/10.1109/TSE.2018.2821670
visibility

…

description

18 pages

link

1 file

Abstract

Transfer learning has been the subject of much recent research. In practice, that research means that the models are unstable since they are continually revised whenever new data arrives. This paper offers a very simple "bellwether" transfer learner. Given N datasets, we find which one produces the best predictions on all the others. This "bellwether" dataset is then used for all subsequent predictions (when its predictions start failing, one may seek another bellwether). Bellwethers are interesting since they are very simple to find (wrap a for-loop around standard data miners). They simplify the task of making general policies in software engineering since as long as one bellwether remains useful, stable conclusions for N datasets can be achieved by reasoning over that bellwether. This paper shows that this bellwether approach works for multiple datasets from various domains in SE. From this, we conclude that (1) bellwether method is a useful (and simple) transfer learner; (2) Unlike bellwethers, other complex transfer learners do not generalized to all domains in SE; (3) "bellwethers" are a baseline method against which future transfer learners should be compared; (4) When building increasingly complex automatic methods, researchers should pause and compare more sophisticated method against simpler alternatives.

Related papers

Limits of Transfer Learning
George Montanez

2020

Transfer learning involves taking information and insight from one problem domain and applying it to a new problem domain. Although widely used in practice, theory for transfer learning remains less well-developed. To address this, we prove several novel results related to transfer learning, showing the need to carefully select which sets of information to transfer and the need for dependence between transferred information and target problems. Furthermore, we prove how the degree of probabilistic change in an algorithm using transfer learning places an upper bound on the amount of improvement possible. These results build on the algorithmic search framework for machine learning, allowing the results to apply to a wide range of learning problems using transfer.

View PDFchevron_right
Distribution-Based Categorization of Classifier Transfer Learning
Jorge Magnun Santos

arXiv (Cornell University), 2017

Transfer Learning (TL) aims to transfer knowledge acquired in one problem, the source problem, onto another problem, the target problem, dispensing with the bottom-up construction of the target model. Due to its relevance, TL has gained significant interest in the Machine Learning (ML) community since it paves the way to devise intelligent learning models that can easily be tailored to many different applications. As it is natural in a fast evolving area, a wide variety of TL methods, settings and nomenclature have been proposed so far. However, a wide range of works have been reporting different names for the same concepts. This concept and terminology mixture contribute however to obscure the TL field, hindering its proper consideration. In this paper we present a review of the literature on the majority of classification TL methods, and also a distribution-based categorization of TL with a common nomenclature suitable to classification problems. Under this perspective three main TL categories are presented, discussed and illustrated with examples.

View PDFchevron_right
Rule Extraction for Transfer Learning
Richard Maclin
View PDFchevron_right
Cross Validation Framework to Choose amongst Models and Datasets for Transfer Learning
Olivier Verscheure

Lecture Notes in Computer Science, 2010

One solution to the lack of label problem is to exploit transfer learning, whereby one acquires knowledge from source-domains to improve the learning performance in the target-domain. The main challenge is that the source and target domains may have different distributions. An open problem is how to select the available models (including algorithms and parameters) and importantly, abundance of source-domain data, through statistically reliable methods, thus making transfer learning practical and easy-to-use for real-world applications. To address this challenge, one needs to take into account the difference in both marginal and conditional distributions in the same time, but not just one of them. In this paper, we formulate a new criterion to overcome "double" distribution shift and present a practical approach "Transfer Cross Validation" (TrCV) to select both models and data in a cross validation framework, optimized for transfer learning. The idea is to use density ratio weighting to overcome the difference in marginal distributions and propose a "reverse validation" procedure to quantify how well a model approximates the true conditional distribution of target-domain. The usefulness of TrCV is demonstrated on different cross-domain tasks, including wine quality evaluation, web-user ranking and text categorization. The experiment results show that the proposed method outperforms both traditional cross-validation and one state-of-the-art method which only considers marginal distribution shift. The software and datasets are available from the authors.

View PDFchevron_right
Machine learning for software engineering: Case studies in software reuse
Tim Menzies

2002

Abstract There are many machine learning algorithms currently available. In the 21st century, the problem no longer lies in writing the learner but in choosing which learners to run on a given data set. We argue that the final choice of learners should not be exclusive; in fact, there are distinct advantages in running data sets through multiple learners. To illustrate our point, we perform a case study on a reuse data set using three different styles of learners: association rule, decision tree induction, and treatment.

View PDFchevron_right
Actively Transfer Domain Knowledge
Xiaoxiao Fan

2008

When labeled examples are not readily available, active learning and transfer learning are separate efforts to obtain labeled examples for inductive learning. Active learning asks domain experts to label a small set of examples, but there is a cost incurred for each answer. While transfer learning could borrow labeled examples from a different domain without incurring any labeling cost, there is no guarantee that the transferred examples will actually help improve the learning accuracy. To solve both problems, we propose a framework to actively transfer the knowledge across domains, and the key intuition is to use the knowledge transferred from other domain as often as possible to help learn the current domain, and query experts only when necessary. To do so, labeled examples from the other domain (out-of-domain) are examined on the basis of their likelihood to correctly label the examples of the current domain (in-domain)

View PDFchevron_right
An experts algorithm for transfer learning
satinder singh

Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, 2007

A long-lived agent continually faces new tasks in its environment. Such an agent may be able to use knowledge learned in solving earlier tasks to produce candidate policies for its current task. There may, however, be multiple reasonable policies suggested by prior experience, and the agent must choose between them potentially without any a priori knowledge about their applicability to its current situation. We present an "experts" algorithm for efficiently choosing amongst candidate policies in solving an unknown Markov decision process task. We conclude with the results of experiments on two domains in which we generate candidate policies from solutions to related tasks and use our experts algorithm to choose amongst them.

View PDFchevron_right
Modeling Transfer Relationships Between Learning Tasks for Improved Inductive Transfer
Marie desJardins

Lecture Notes in Computer Science, 2008

In this paper, we propose a novel graph-based method for knowledge transfer. We model the transfer relationships between source tasks by embedding the set of learned source models in a graph using transferability as the metric. Transfer to a new problem proceeds by mapping the problem into the graph, then learning a function on this graph that automatically determines the parameters to transfer to the new learning task. This method is analogous to inductive transfer along a manifold that captures the transfer relationships between the tasks. We demonstrate improved transfer performance using this method against existing approaches in several real-world domains.

View PDFchevron_right
Guided Transfer Learning
Davor Andric

arXiv (Cornell University), 2023

Machine learning requires exuberant amounts of data and computation. Also, models require equally excessive growth in the number of parameters. It is, therefore, sensible to look for technologies that reduce these demands on resources. Here, we propose an approach called guided transfer learning. Each weight and bias in the network has its own guiding parameter that indicates how much this parameter is allowed to change while learning a new task. Guiding parameters are learned during an initial scouting process. Guided transfer learning can result in a reduction in resources needed to train a network. In some applications, guided transfer learning enables the network to learn from a small amount of data. In other cases, a network with a smaller number of parameters can learn a task which otherwise only a larger network could learn. Guided transfer learning potentially has many applications when the amount of data, model size, or the availability of computational resources reach their limits.

View PDFchevron_right
The Case for Case-Based Transfer Learning
Matthew Molineaux

AI Magazine, 2011

Case-based reasoning (CBR) is a problem-solving process in which a new problem is solved by retrieving a similar situation and reusing its solution. Transfer learning occurs when, after gaining experience from learning how to solve source problems, the same learner exploits this experience to improve performance and/or learning on target problems. In transfer learning, the differences between the source and target problems characterize the transfer distance. CBR can support transfer learning methods in multiple ways. We illustrate how CBR and transfer learning interact and characterize three approaches for using CBR in transfer learning: (1) as a transfer learning method, (2) for problem learning, and (3) to transfer knowledge between sets of problems. We describe examples of these approaches from our own and related work and discuss applicable transfer distances for each. We close with conclusions and directions for future research applying CBR to transfer learning.

View PDFchevron_right

References (84)

  1. RQ2: How well do transfer learners perform across different domains? RQ4: How much data is required to find the bellwether dataset? REFERENCES
  2. J. Czerwonka, R. Das, N. Nagappan, A. Tarvo, and A. Teterev, "Crane: Failure prediction, change analysis and test prioritization in practice -experiences from windows," in Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth International Conference on, march 2011, pp. 357 -366.
  3. T. J. Ostrand, E. J. Weyuker, and R. M. Bell, "Where the bugs are," in ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis. New York, NY, USA: ACM, 2004, pp. 86-96.
  4. T. Menzies, A. Dekhtyar, J. Distefano, and J. Greenwald, "Problems with Precision: A Response to "Comments on 'Data Mining Static Code Attributes to Learn Defect Predictors'"," IEEE Transactions on Software Engineering, vol. 33, no. 9, pp. 637-640, sep 2007. [Online]. Available: http://ieeexplore.ieee.org/ lpdocs/epic03/wrapper.htm?arnumber=4288197
  5. B. Turhan, A. Tosun, and A. Bener, "Empirical evaluation of mixed-project defect prediction models," in Software Engineering and Advanced Applications (SEAA), 2011 37th EUROMICRO Conference on. IEEE, 2011, pp. 396-403.
  6. E. Kocaguneli, T. Menzies, A. Bener, and J. Keung, "Exploiting the essential assumptions of analogy-based effort estimation," IEEE Transactions on Software Engineering, vol. 28, pp. 425-438, 2012, available from http://menzies.us/pdf/ 11teak.pdf.
  7. T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck, and M. Stumm, "Contin- uous deployment at facebook and oanda," in Proceedings of the 38th International Conference on Software Engineering Companion. ACM, 2016, pp. 21-30.
  8. M. Linares-Vásquez, G. Bavota, C. Bernal-Cárdenas, R. Oliveto, M. Di Penta, and D. Poshyvanyk, "Mining energy-greedy api usage patterns in android apps: an empirical study," in Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 2014, pp. 2-11.
  9. A. Begel and T. Zimmermann, "Analyze this! 145 questions for data scientists in software engineering," in Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, June 2014. [Online]. Available: http://research.microsoft.com/apps/pubs/default.aspx?id=208800
  10. C. Theisen, K. Herzig, P. Morrison, B. Murphy, and L. Williams, "Approximating attack surfaces with stack traces," in ICSE'15, 2015.
  11. T. Zimmermann and T. Menzies, "Software analytics: So what?" IEEE Software, vol. 30, no. 4, pp. 0031-37, 2013.
  12. C. Bird, T. Menzies, and T. Zimmermann, The Art and Science of Analyzing Software Data. Elsevier, 2015.
  13. F. Sarro, A. Petrozziello, and M. Harman, "Multi-objective software effort estima- tion," in 38th International Conference on Software Engineering (ICSE'16), no. to appear. ACM, 2016.
  14. M. Choetkiertikul, H. K. Dam, T. Tran, and A. Ghose, "Predicting delays in software projects using networked classification (t)," in Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 2015, pp. 353-364.
  15. F. Rahman, S. Khatri, E. T. Barr, and P. Devanbu, "Comparing static bug finders and statistical prediction," in Proceedings of the 36th International Conference on Software Engineering. ACM, 2014, pp. 424-434.
  16. Y. Shin and L. Williams, "Can traditional fault prediction models be used for vulnerability prediction?" Empirical Software Engineering, vol. 18, no. 1, pp. 25- 59, 2013.
  17. B. Turhan, T. Menzies, A. Bener, and J. Distefano, "On the relative value of cross-company and within-company data for defect prediction," Empirical Software Engineering, vol. 68, no. 2, pp. 278-290, 2009, available from http://menzies.us/ pdf/08ccwc.pdf.
  18. J. Nam and S. Kim, "Heterogeneous defect prediction," in Proc. 2015 10th Jt. Meet. Found. Softw. Eng. -ESEC/FSE 2015. New York, New York, USA: ACM Press, 2015, pp. 508-519. [Online]. Available: http: //dl.acm.org/citation.cfm?doid=2786805.2786814
  19. Z. He, F. Peters, T. Menzies, and Y. Yang, "Learning from open-source projects: An empirical study on defect prediction," in Empirical Software Engineering and Measurement, 2013 ACM/IEEE International Symposium on. IEEE, 2013, pp. 45-54.
  20. A. Tosun, A. Bener, and R. Kale, "AI-based software defect predictors: Applications and benefits in a case study," in Twenty-Second IAAI Conference on Artificial Intelligence, 2010.
  21. A. Tosun, A. Bener, and B. Turhan, "Practical considerations of deploying ai in defect prediction: A case study within the Turkish telecommunication industry," in PROMISE'09, 2009.
  22. M. Lumpe, R. Vasa, T. Menzies, R. Rush, and R. Turhan, "Learning better inspection optimization policies," International Journal of Software Engineering and Knowledge Engineering, vol. 21, no. 45, pp. 725-753, 2011.
  23. C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu, "Fair and balanced?: bias in bug-fix datasets," in Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 2009, pp. 121-130.
  24. F. Rahman and P. Devanbu, "How, and why, process metrics are better," in 2013 35th International Conference on Software Engineering (ICSE), May 2013, pp. 432-441.
  25. J. Nam, S. J. Pan, and S. Kim, "Transfer defect learning," in Proceedings - International Conference on Software Engineering, 2013, pp. 382-391.
  26. X. Jing, F. Wu, X. Dong, F. Qi, and B. Xu, "Heterogeneous Cross-Company Defect Prediction by Unified Metric Representation and CCA-Based Transfer Learning Categories and Subject Descriptors," Proceeding of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2015), pp. 496-507, 2015.
  27. E. Kocaguneli and T. Menzies, "How to find relevant data for effort estimation?" in Empirical Software Engineering and Measurement (ESEM), 2011 International Symposium on. IEEE, 2011, pp. 255-264.
  28. E. Kocaguneli, T. Menzies, and E. Mendes, "Transfer learning in effort estimation," Empirical Software Engineering, vol. 20, no. 3, pp. 813-843, jun 2015. [Online]. Available: http://link.springer.com/10.1007/s10664-014-9300-5
  29. B. Turhan, T. Menzies, A. B. Bener, and J. Di Stefano, "On the relative value of cross-company and within-company data for defect prediction," Empirical Software Engineering, vol. 14, no. 5, pp. 540-578, 2009.
  30. F. Peters, T. Menzies, and L. Layman, "LACE2: Better privacy-preserving data sharing for cross project defect prediction," in Proceedings -International Confer- ence on Software Engineering, vol. 1, 2015, pp. 801-811.
  31. F. Rahman, D. Posnett, and P. Devanbu, "Recalling the "imprecision" of cross-project defect prediction," in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, ser. FSE '12. New York, NY, USA: ACM, 2012, pp. 61:1-61:11. [Online]. Available: http://doi.acm.org/10.1145/2393596.2393669
  32. R. Krishna, T. Menzies, and W. Fu, "Too Much Automation? The Bellwether Effect and Its Implications for Transfer Learning," in ASE'16, 2016.
  33. Y. Ma, G. Luo, X. Zeng, and A. Chen, "Transfer learning for cross-company software defect prediction," Information and Software Technology, vol. 54, no. 3, pp. 248-256, 2012.
  34. D. Ryu, O. Choi, and J. Baik, "Value-cognitive boosting with a support vector machine for cross-project defect prediction," Empir. Softw. Eng., vol. 21, no. 1, pp. 43-71, feb 2016. [Online]. Available: http://link.springer.com/10.1007/ s10664-014-9346-4
  35. T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, "Cross-project defect prediction: a large scale experiment on data vs. domain vs. process," in Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 2009, pp. 91-100.
  36. F. Zhang, A. Mockus, I. Keivanloo, and Y. Zou, "Towards building a universal defect prediction model with rank transformed predictors," Empirical Software Engineering, pp. 1-39, 2015. [Online]. Available: http: //dx.doi.org/10.1007/s10664-015-9396-2
  37. T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell, "A systematic literature review on fault prediction performance in software engineering," IEEE Transactions on Software Engineering, vol. 38, no. 6, pp. 1276-1304, Nov 2012.
  38. R. Krishna, T. Menzies, and L. Layman, "Less is More: Minimizing Code Reorganization using XTREE," CoRR, vol. abs/1609.03614, 2016. [Online]. Available: http://arxiv.org/abs/1609.03614
  39. M. Fowler, K. Beck, J. Brant, W. Opdyke, and D. Roberts, Refactoring: Improving the Design of Existing Code. Boston, MA, USA: Addison-Wesley Longman, 1999.
  40. A. Yamashita and S. Counsell, "Code smells as system-level indicators of main- tainability: An empirical study," Journal of Systems and Software, vol. 86, no. 10, pp. 2639-2653, 2013.
  41. A. Yamashita and L. Moonen, "Exploring the impact of inter-smell relations on software maintainability: An empirical study," in Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 2013, pp. 682- 691.
  42. N. Zazworka, M. A. Shaw, F. Shull, and C. Seaman, "Investigating the impact of design debt on software quality," in Proceedings of the 2nd Workshop on Managing Technical Debt. ACM, 2011, pp. 17-23.
  43. F. A. Fontana, P. Braione, and M. Zanoni, "Automatic detection of bad smells in code: An experimental assessment." Journal of Object Technology, vol. 11, no. 2, pp. 5-1, 2012.
  44. J. Kreimer, "Adaptive Detection of Design Flaws," Electronic Notes in Theoretical Computer Science, vol. 141, no. 4, pp. 117-136, 2005. [Online]. Available: //www.sciencedirect.com/science/article/pii/S1571066105051844
  45. F. Khomh, S. Vaucher, Y. G. Guhneuc, and H. Sahraoui, "A bayesian approach for the detection of code and design smells," in 2009 Ninth International Conference on Quality Software, Aug 2009, pp. 305-314.
  46. F. Khomh, S. Vaucher, Y.-G. Guhneuc, and H. Sahraoui, "Bdtex: A gqm-based bayesian approach for the detection of antipatterns," Journal of Systems and Software, vol. 84, no. 4, pp. 559 -572, 2011, the Ninth International Conference on Quality Software. [Online]. Available: //www.sciencedirect.com/science/article/pii/S0164121210003225
  47. J. Yang, K. Hotta, Y. Higo, H. Igaki, and S. Kusumoto, "Filtering clones for individual user based on machine learning analysis," in 2012 6th International Workshop on Software Clones (IWSC), June 2012, pp. 76-77.
  48. F. Arcelli Fontana, M. V. Mäntylä, M. Zanoni, and A. Marino, "Comparing and experimenting machine learning techniques for code smell detection," Empir. Softw. Eng., vol. 21, no. 3, pp. 1143-1191, jun 2016. [Online]. Available: http://dx.doi.org/ 10.1007/s10664-015-9378-4http://link.springer.com/10.1007/s10664-015-9378-4
  49. E. Tempero, C. Anslow, J. Dietrich, T. Han, J. Li, M. Lumpe, H. Melton, and J. Noble, "Qualitas corpus: A curated collection of java code for empirical studies," in 2010 Asia Pacific Software Engineering Conference (APSEC2010), Dec. 2010, pp. 336-345.
  50. L. D. Panjer, "Predicting eclipse bug lifetimes," in Proceedings of the Fourth International Workshop on Mining Software Repositories, ser. MSR '07. Washington, DC, USA: IEEE Computer Society, 2007, pp. 29-. [Online]. Available: http://dx.doi.org/10.1109/MSR.2007.25
  51. E. Giger, M. Pinzger, and H. Gall, "Predicting the fix time of bugs," in Proceedings of the 2Nd International Workshop on Recommendation Systems for Software Engineering, ser. RSSE '10. New York, NY, USA: ACM, 2010, pp. 52-56. [Online]. Available: http://doi.acm.org/10.1145/1808920.1808933
  52. H. Zhang, L. Gong, and S. Versteeg, "Predicting bug-fixing time: An empirical study of commercial software projects," in Proceedings of the 2013 International Conference on Software Engineering, ser. ICSE '13. Piscataway, NJ, USA: IEEE Press, 2013, pp. 1042-1051. [Online]. Available: http://dl.acm.org/citation.cfm?id=2486788.2486931
  53. M. Rees-jones, M. Martin, C. College, and T. Menzies, "Better Predictors for Issue Lifetime," , pp. 1-8, 2017.
  54. R. Kikas, M. Dumas, and D. Pfahl, "Using dynamic and contextual features to predict issue lifetime in github projects," in Proceedings of the 13th International Conference on Mining Software Repositories, ser. MSR '16. New York, NY, USA: ACM, 2016, pp. 291-302. [Online]. Available: http://doi.acm.org/10.1145/2901739.2901751
  55. B. A. Kitchenham, E. Mendes, and G. H. Travassos, "Cross versus within-company cost estimation studies: A systematic review," IEEE Trans. Softw. Eng., vol. 33, no. 5, pp. 316-329, May 2007. [Online]. Available: http://dx.doi.org/10.1109/TSE.2007.1001
  56. Y. Yang, L. Xie, Z. He, Q. Li, V. Nguyen, B. Boehm, and R. Valerdi, "Local bias and its impacts on the performance of parametric estimation models," in Proceedings of the 7th International Conference on Predictive Models in Software Engineering -Promise '11. New York, New York, USA: ACM Press, 2011, pp. 1-10. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2020390.2020404
  57. T. Menzies, Y. Yang, G. Mathew, B. Boehm, and J. Hihn, "Negative results for software effort estimation," Empirical Software Engineering, pp. 1-26, 2016. [Online]. Available: http://dx.doi.org/10.1007/s10664-016-9472-2
  58. B. W. Boehm et al., Software engineering economics. Prentice-hall Englewood Cliffs (NJ), 1981, vol. 197.
  59. S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, "Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings," IEEE Trans. Softw. Eng., vol. 34, no. 4, pp. 485-496, jul 2008. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm? arnumber=4527256
  60. M. Lowry, M. Boyd, and D. Kulkami, "Towards a theory for integration of math- ematical verification and empirical testing," in Automated Software Engineering, 1998. Proceedings. 13th IEEE International Conference on. IEEE, 1998, pp. 322-331.
  61. N. Nagappan and T. Ball, "Static analysis tools as early indicators of pre-release defect density," in ICSE 2005, St. Louis, 2005.
  62. T. Menzies, D. Raffo, S. on Setamanit, Y. Hu, and S. Tootoonian, "Model-based tests of truisms," in Proceedings of IEEE ASE 2002, 2002, available from http: //menzies.us/pdf/02truisms.pdf.
  63. C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead Jr., "Does bug prediction support human developers? findings from a google case study," in Proceedings of the 2013 International Conference on Software Engineering, ser. ICSE '13. Piscataway, NJ, USA: IEEE Press, 2013, pp. 372-381. [Online]. Available: http://dl.acm.org/citation.cfm?id=2486788.2486838
  64. S. Rakitin, Software Verification and Validation for Practitioners and Managers, Second Edition. Artech House, 2001.
  65. T. Menzies, J. Greenwald, and A. Frank, "Data mining static code attributes to learn defect predictors," IEEE Transactions on Software Engineering, January 2007, available from http://menzies.us/pdf/06learnPredict.pdf.
  66. F. Shull, V. B. ad B. Boehm, A. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R. Tesoriero, and M. Zelkowitz, "What we have learned about fighting defects," in Proceedings of 8th International Software Metrics Symposium, Ottawa, Canada, 2002, pp. 249-258.
  67. M. Fagan, "Design and code inspections to reduce errors in program development," IBM Systems Journal, vol. 15, no. 3, 1976.
  68. M. D'Ambros, M. Lanza, and R. Robbes, "Evaluating defect prediction approaches: a benchmark and an extensive comparison," Empir. Softw. Eng., vol. 17, no. 4-5, pp. 531-577, aug 2012. [Online]. Available: http: //link.springer.com/10.1007/s10664-011-9173-9
  69. R. Wu, H. Zhang, S. Kim, and S.-C. Cheung, "ReLink," in Proc. 19th ACM SIGSOFT Symp. 13th Eur. Conf. Found. Softw. Eng. -SIGSOFT/FSE '11. New York, New York, USA: ACM Press, 2011, p. 15.
  70. V. R. Basili, L. C. Briand, and W. L. Melo, "A validation of object-oriented design metrics as quality indicators," Software Engineering, IEEE Transactions on, vol. 22, no. 10, pp. 751-761, 1996.
  71. N. Ohlsson and H. Alberg, "Predicting fault-prone software modules in telephone switches," Software Engineering, IEEE Transactions on, vol. 22, no. 12, pp. 886- 894, 1996.
  72. S. Kim, H. Zhang, R. Wu, and L. Gong, "Dealing with noise in defect prediction," in Software Engineering (ICSE), 2011 33rd International Conference on. IEEE, 2011, pp. 481-490.
  73. M. Jureczko and L. Madeyski, "Towards identifying software project clusters with regard to defect prediction," in Proc. 6th Int. Conf. Predict. Model. Softw. Eng. - PROMISE '10. New York, New York, USA: ACM Press, 2010, p. 1. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1868328.1868342
  74. L. Breiman, "Random forests," Machine learning, pp. 5-32, 2001. [Online]. Available: http://link.springer.com/article/10.1023/A:1010933404324
  75. L. Pelayo and S. Dick, "Applying Novel Resampling Strategies To Software Defect Prediction," in NAFIPS 2007 -2007 Annu. Meet. North Am. Fuzzy Inf. Process. Soc. IEEE, jun 2007, pp. 69-72. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4271036
  76. N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, "SMOTE: Synthetic minority over-sampling technique," J. Artif. Intell. Res., vol. 16, 2002.
  77. Y. Ma and B. Cukic, "Adequate and precise evaluation of quality models in software engineering studies," in Predictor Models in Software Engineering, 2007. PROMISE'07: ICSE Workshops 2007. International Workshop on, May 2007, pp. 1-1.
  78. N. Mittas and L. Angelis, "Ranking and clustering software cost estimation models through a multiple comparisons algorithm," IEEE Trans. Software Eng., vol. 39, no. 4, pp. 537-551, 2013.
  79. B. Efron and R. J. Tibshirani, An introduction to the bootstrap, ser. Mono. Stat. Appl. Probab. London: Chapman and Hall, 1993.
  80. M. J. Shepperd and S. G. MacDonell, "Evaluating prediction systems in software project estimation," Information & Software Technology, vol. 54, no. 8, pp. 820- 827, 2012.
  81. V. B. Kampenes, T. Dybå, J. E. Hannay, and D. I. K. Sjøberg, "A systematic review of effect size in software engineering experiments," Information & Software Technology, vol. 49, no. 11-12, pp. 1073-1086, 2007.
  82. E. Kocaguneli, T. Zimmermann, C. Bird, N. Nagappan, and T. Menzies, "Dis- tributed development considered harmful?" in Proceedings -International Confer- ence on Software Engineering, 2013, pp. 882-890.
  83. A. Arcuri and L. Briand, "A practical guide for using statistical tests to assess randomized algorithms in software engineering," in ICSE'11, 2011, pp. 1-10.
  84. W. Fu, T. Menzies, and X. Shen, "Tuning for software analytics: Is it really necessary?" Information and Software Technology, vol. 76, pp. 135 - 146, 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/ S0950584916300738

Related papers

Simpler Transfer Learning (Using "Bellwethers")
Tim Menzies

ArXiv, 2017

Transfer learning has been the subject of much recent research. In practice, that research means that the models are unstable since they are continually revised whenever new data arrives. This paper offers a very simple “bellwether” transfer learner. Given N datasets, we find which one produces the best predictions on all the others. This “bellwether” dataset is then used for all subsequent predictions (when its predictions start failing, one may seek another bellwether). Bellwethers are interesting since they are very simple to find (wrap a for-loop around standard data miners). They simplify the task of making general policies in software engineering since as long as one bellwether remains useful, stable conclusions for N datasets can be achieved by reasoning over that bellwether. This paper shows that this bellwether approach works for multiple datasets from various domains in SE. From this, we conclude that (1) bellwether method is a useful (and simple) transfer learner; (2) Unl...

View PDFchevron_right
Transfer Learning for Improving Model Predictions in Highly Configurable Software
Miguel Velez

2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), 2017

Modern software systems are built to be used in dynamic environments using configuration capabilities to adapt to changes and external uncertainties. In a self-adaptation context, we are often interested in reasoning about the performance of the systems under different configurations. Usually, we learn a black-box model based on real measurements to predict the performance of the system given a specific configuration. However, as modern systems become more complex, there are many configuration parameters that may interact and we end up learning an exponentially large configuration space. Naturally, this does not scale when relying on real measurements in the actual changing environment. We propose a different solution: Instead of taking the measurements from the real system, we learn the model using samples from other sources, such as simulators that approximate performance of the real system at low cost. We define a cost model that transform the traditional view of model learning into a multi-objective problem that not only takes into account model accuracy but also measurements effort as well. We evaluate our cost-aware transfer learning solution using real-world configurable software including (i) a robotic system, (ii) 3 different stream processing applications, and (iii) a NoSQL database system. The experimental results demonstrate that our approach can achieve (a) a high prediction accuracy, as well as (b) a high model reliability.

View PDFchevron_right
A Concise Review of Transfer Learning
Hamid Arabnia

2020 International Conference on Computational Science and Computational Intelligence (CSCI), 2020

The availability of abundant labeled data in recent years led the researchers to introduce a methodology called transfer learning, which utilizes existing data in situations where there are difficulties in collecting new annotated data. Transfer learning aims to boost the performance of a target learner by applying another related source data. In contrast to the traditional machine learning and data mining techniques, which assume that the training and testing data lie from the same feature space and distribution, transfer learning can handle situations where there is a discrepancy between domains and distributions. These characteristics give the model the potential to utilize the available related source data and extend the underlying knowledge to the target task achieving better performance. This survey paper aims to give a concise review of traditional and current transfer learning settings, existing challenges, and related approaches.

View PDFchevron_right
Transfer learning: a friendly introduction
Asmaul Hosna

Journal of Big Data

Infinite numbers of real-world applications use Machine Learning (ML) techniques to develop potentially the best data available for the users. Transfer learning (TL), one of the categories under ML, has received much attention from the research communities in the past few years. Traditional ML algorithms perform under the assumption that a model uses limited data distribution to train and test samples. These conventional methods predict target tasks undemanding and are applied to small data distribution. However, this issue conceivably is resolved using TL. TL is acknowledged for its connectivity among the additional testing and training samples resulting in faster output with efficient results. This paper contributes to the domain and scope of TL, citing situational use based on their periods and a few of its applications. The paper provides an in-depth focus on the techniques; Inductive TL, Transductive TL, Unsupervised TL, which consists of sample selection, and domain adaptation...

View PDFchevron_right
Effecting Transfer via Learning Curve Analysis
Christophe Giraud-Carrier

2005

Abstract We consider the idea of using learning curves to characterize learning tasks and ask the question of whether such a characterization is useful for inductive transfer. We compute a database of learning curves for 49 datasets and 9 learning algorithms and demonstrate its utility in transferring information about expected performance.

View PDFchevron_right

Related topics

  • Information Systems

    • Cited by

    Revisiting process versus product metrics: a large scale analysis
    Suvodeep Majumder

    Empirical Software Engineering

    Numerous methods can build predictive models from software data. However, what methods and conclusions should we endorse as we move from analytics in-the-small (dealing with a handful of projects) to analytics in-thelarge (dealing with hundreds of projects)? To answer this question, we recheck prior small-scale results (about process versus product metrics for defect prediction and the granularity of metrics) using 722,471 commits from 700 Github projects. We find that some analytics in-the-small conclusions still hold when scaling up to analytics in-the-large. For example, like prior work, we see that process metrics are better predictors for defects than product metrics (best process/product-based learners respectively achieve recalls of 98%/44% and AUCs of 95%/54%, median values). That said, we warn that it is unwise to trust metric importance results from analytics in-the-small studies since those change dramatically when moving to analytics in-the-large. Also, when reasoning in-the-large about hundreds of projects, it is better to use predictions from multiple models (since single model predictions can become confused and exhibit a high variance).

    View PDFchevron_right
    Learning actionable analytics from multiple software projects
    Tim Menzies

    Empirical Software Engineering, 2020

    The current generation of software analytics tools are mostly prediction algorithms (e.g. support vector machines, naive bayes, logistic regression, etc). While prediction is useful, after prediction comes planning about what actions to take in order to improve quality. This research seeks methods that generate demonstrably useful guidance on "what to do" within the context of a specific software project. Specifically, we propose XTREE (for within-project planning) and BELLTREE (for cross-project planning) to generating plans that can improve software quality. Each such plan has the property that, if followed, it reduces the expected number of future defect reports. To find this expected number, planning was first applied to data from release x. Next, we looked for change in release x + 1 that conformed to our plans. This procedure was applied using a range of planners from the literature, as well as XTREE. In 10 open-source JAVA systems, several hundreds of defects were reduced in sections of the code that conformed to XTREE's plans. Further, when compared to other planners, XTREE's plans were found to be easier to implement (since they were shorter) and more effective at reducing the expected number of defects.

    View PDFchevron_right
    Questions for data scientists in software engineering: a replication
    Hennie Huijgens

    Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020

    In 2014, a Microsoft study investigated the sort of questions that data science applied to software engineering should answer. This resulted in 145 questions that developers considered relevant for data scientists to answer, thus providing a research agenda to the community. Fast forward to five years, no further studies investigated whether the questions from the software engineers at Microsoft hold for other software companies, including software-intensive companies with different primary focus (to which we refer as software-defined enterprises). Furthermore, it is not evident that the problems identified five years ago are still applicable, given the technological advances in software engineering. This paper presents a study at ING, a software-defined enterprise in banking in which over 15,000 IT staff provides in-house software solutions. This paper presents a comprehensive guide of questions for data scientists selected from the previous study at Microsoft along with our current work at ING. We replicated the original Microsoft study at ING, looking for questions that impact both software companies and software-defined enterprises and continue to impact software engineering. We also add new questions that emerged from differences in the context of the two companies and the five years gap in between. Our results show that software engineering questions for data scientists in the software-defined enterprise are largely similar to the software company, albeit with exceptions. We hope that the software engineering research community builds on the new list of questions to create a useful body of knowledge.

    View PDFchevron_right
    Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers
    Markus Wagner

    Empirical Software Engineering, 2020

    This paper claims that a new field of empirical software engineering research and practice is emerging: data mining using/used-by optimizers for empirical studies, or DUO. For example, data miners can generate models that are explored by optimizers. Also, optimizers can advise how to best adjust the control parameters of a data miner. This combined approach acts like an agent leaning over the shoulder of an analyst that advises "ask this question next" or "ignore that problem, it is not relevant to your goals". Further, those agents can help us build "better" predictive models, where "better" can be either greater predictive accuracy or faster modeling time (which, in turn, enables the exploration of a wider range of options). We also caution that the era of papers that just use data miners is coming to an end. Results obtained from an unoptimized data miner can be quickly refuted, just by applying an optimizer to produce a different (and better performing) model. Our conclusion, hence, is that for software analytics it is possible, useful and necessary to combine data mining and optimization using DUO.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved