Academia.eduAcademia.edu

Outline

Reproducibility in the technical debt domain

2022, Acta Universitatis Sapientiae, Informatica

https://doi.org/10.2478/AUSI-2021-00016

Abstract

Context: It is crucial to understand how reproducible the measurement results in the scientific publications are, as reproducibility is one of the cornerstones of engineering. Objective: The goal of this study is to investigate the scientific publications presented at the premier technical debt conferences by understanding how reproducible the reported findings are. Method: We conducted a systematic literature review of 135 unique papers published at the "International Workshop on Managing Technical Debt" and the "International Conference on Managing Technical Debt", the premier scientific conference series on technical debt. Results: Only 44 of the investigated 135 papers presented numerical evidence and only 5 papers listed the tools, the availability of the tools, and the version of the tools used. For the rest of the papers additional information would have been needed for the potential reproducibility. One of the published papers even referred to a pornographic site as a source of a toolset for empirical research.

References (72)

  1. S. Ahmadjee, C. Mera-Gómez, R. Bahsoon, Assessing Smart Contracts Security Technical Debts, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 6-15. ⇒ 350
  2. A. Aldaeej, C. Seaman, From Lasagna to Spaghetti: A Decision Model to Man- age Defect Debt, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 67-71. ⇒ 348
  3. R. Alfayez, W. Alwehaibi, R. Winn, E. Venson, B. Boehm, A systematic liter- ature review of technical debt prioritization, In Proceedings of the 3rd Interna- tional Conference on Technical Debt (TechDebt '20), 2020, pp. 1-10. ⇒ 341
  4. R. R. Alfayez, P. Behnamghader, K. Srisopha, B. Boehm, An Exploratory Study on the Influence of Developers in Technical Debt, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 1-10. ⇒ 348
  5. S. Alhusain, Predicting Relative Thresholds for Object Oriented Metrics, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 55-63. ⇒ 351
  6. T. Amanatidis, N. Mittas, A. Chatzigeorgiou, A. Ampatzoglou, L. Angelis, The Developer's Dilemma: Factors Affecting the Decision to Repay Code Debt, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 62-66. ⇒ 348
  7. A. Ampatzoglou, A. Ampatzoglou, A. Chatzigeorgiou, P. Avgeriou, P. Abra- hamsson, A. Martini, U. Zdun, K. Systa, The Perception of Technical Debt in the Embedded Systems Domain: An Industrial Case Study, 2016 IEEE 8th In- ternational Workshop on Managing Technical Debt (MTD), 2016, pp. 9-16. ⇒ 347
  8. A. Ampatzoglou, A. Michailidis, C. Sarikyriakidis, A. Ampatzoglou, A. Chatzi- georgiou, P. Avgeriou, A Framework for Managing Interest in Technical Debt: An Industrial Validation, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 115-124. ⇒ 337, 347, 349, 352
  9. M. F. Aniche, G. A. Oliva, M. A. Gerosa, Are the Methods in Your Data Access Objects (DAOs) in the Right Place? A Preliminary Study, Sixth International Workshop on Managing Technical Debt (MTD '14), 2014, pp. 47-50. ⇒ 345
  10. P. C. Avgeriou, D. Taibi, A. Ampatzoglou, A. F. Fontana, T. Besker, A. Chatzi- georgiou, V. Lenarduzzi, A. Martini, A. Moschou, I. Pigazzini, N. Saarimäki, D. Sas, S. Toledo, A. Tsintzira. An Overview and Comparison of Technical Debt Measurement Tools, in IEEE Software, vol. 38, no. 3, pp. 61-71, May-June 2021. ⇒ 336, 338
  11. B. Barta, G. Manz, I. Siket, R. Ferenc, Challenges of SonarQube Plug-In Main- tenance, IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2019, pp. 574-578. ⇒ 336, 338
  12. T. Besker, A. Martini, J. Bosch, Technical debt cripples software developer pro- ductivity: a longitudinal study on developers' daily software development work, In Proceedings of the 2018 IEEE/ACM International Conference on Technical Debt (TechDebt '18), 2018, pp. 105-114. ⇒ 341
  13. J. Bohnet, J. Döllner, Monitoring code quality and development activity by software maps, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), ACM, 2011, New York, NY, USA, 9-16. ⇒ 341
  14. N. Brown, Y.g Cai, Y. Guo, R. Kazman, M. Kim, P. Kruchten, E. Lim, A. MacCormack, R. L. Nord, I. Ozkaya, R. Sangwan, C. Seaman, K. Sullivan, N. Zazworka, Managing technical debt in software-reliant systems, In Proceedings of the FSE/SDP workshop on Future of software engineering research (FoSER '10), 2010, pp. 47-52. ⇒ 337, 338, 341, 342, 343
  15. R. Capilla, T. Mikkonen, C. Carrillo, F. A. Fontana, I. Pigazzini, V. Lenarduzzi, Impact of Opportunistic Reuse Practices to Technical Debt, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 16-25. ⇒351
  16. A. Chatzigeorgiou, A. Ampatzoglou, A. Ampatzoglou, T. Amanatidis, Estimat- ing the breaking point for technical debt, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 53-56. ⇒ 346
  17. Carnegie Mellon University, Software Engineering Institute, Managing Technical Debt Workshops, previous editions, 2010, https://resources.sei.cmu.edu/news- events/events/techdebt/past.cfm, last visited: 2021.10.05. ⇒ 342, 345
  18. Y. Crespo, A. Gonzalez-Escribano, M. Piattini, Carrot and Stick approaches revisited when managing Technical Debt in an educational context, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 99-108. ⇒ 351
  19. D. Cruz, A. Santana, E. Figueiredo, Detecting bad smells with machine learning algorithms: an empirical study, In Proceedings of the 3rd International Confer- ence on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 31-40. ⇒ 350
  20. W. Cunningham, The WyCash Portfolio Management System, In Addendum to the proceedings on Object-oriented programming systems, languages, and appli- cations (Addendum) (OOPSLA '92). ACM, NY, USA, 1992, 29-30. ⇒ 337
  21. B. Curtis, J. Sappidi, A. Szynkarski, Estimating the size, cost, and types of Technical Debt, Third International Workshop on Managing Technical Debt (MTD'12), 2012, pp. 49-53. ⇒ 344
  22. R. R. de Almeida, R. do Nascimento Ribeiro, C. Treude, U. Kulesza, Business-Driven Technical Debt Prioritization: An Industrial Case Study, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 74-83. ⇒ 351
  23. M. A. de Freitas Farias, M. G. de Mendonça Neto, A. B. d. Silva, R. O. Spínola, A Contextualized Vocabulary Model for identifying technical debt on code com- ments, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 25-32. ⇒ 346
  24. J. de Groot, A. Nugroho, T. Bäck, J. Visser, What is the value of your software?, Third International Workshop on Managing Technical Debt (MTD'12), 2012, pp. 37-44. ⇒ 344
  25. M. di Biase, A. Rastogi, M. Bruntink, A. van Deursen, The Delta Maintain- ability Model: Measuring Maintainability of Fine-Grained Code Changes, 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, pp. 113-122. ⇒ 349
  26. D. Falessi, R. Kazman, Worst Smells and Their Worst Reasons, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 45-54. ⇒351
  27. D. Falessi, P. Kruchten, R. L. Nord, I. Ozkaya, Technical debt at the crossroads of research and practice: report on the fifth international workshop on managing technical debt, 2014, SIGSOFT Softw. Eng. Notes 39, 2, 31-33. ⇒ 343, 345
  28. D. Falessi, A. Reichel, Towards an open-source tool for measuring and visual- izing the interest of technical debt, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 1-8. ⇒ 346
  29. D. Falessi, A. Voegele, Validating and prioritizing quality rules for managing technical debt: An industrial case study, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 41-48. ⇒ 347
  30. F. A. Fontana, V. Ferme, S. Spinelli, Investigating the impact of code smells debt on quality code evaluation, Third International Workshop on Managing Technical Debt (MTD'12), 2012, pp. 15-22. ⇒ 344
  31. F. A. Fontana, V. Ferme, M. Zanoni, R. Roveda, Towards a prioritization of code debt: A code smell Intensity Index, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 16-24. ⇒ 346
  32. F. A. Fontana, R. Roveda, M. Zanoni, Technical Debt Indexes Provided by Tools: A Preliminary Discussion, 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), 2016, pp. 28-31. ⇒ 347
  33. I. Gat, J. D. Heintz, From assessment to reduction: how cutter consortium helps rein in millions of dollars in technical debt, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, Waikiki, Honolulu, HI, USA, 24-26. ⇒ 343
  34. R. Gomes, C. Siebra, G. Tonin, A. Cavalcanti, Fabio Q.B. da Silva, Andre L.M. Santos, R. Marques, An extraction method to collect data on defects and effort evolution in a constantly modified system, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, Waikiki, Honolulu, HI, USA, 27-30. ⇒ 343
  35. I. Griffith, D. Reimanis, C. Izurieta, Z. Codabux, A. Deo, B. Williams, The Cor- respondence Between Software Quality Models and Technical Debt Estimation Approaches, Sixth International Workshop on Managing Technical Debt (MTD '14), 2014, pp. 19-26. ⇒ 345
  36. I. Khomyakov, Z. Makhmutov, R. Mirgalimova, A. Sillitti, 2019, Automated Measurement of Technical Debt: A Systematic Literature Review, In Proceed- ings of the 21st International Conference on Enterprise Information Systems - Volume 2(ICEIS), 2019, pages 95-106. ⇒ 338
  37. J. R. Lahti, A. -P. Tuovinen, T. Mikkonen, Experiences on Managing Techni- cal Debt with Code Smells and AntiPatterns, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 36-44. ⇒ 351
  38. V. Lenarduzzi, T. Besker, D. Taibi, A. Martini, F. A. Fontana, Techni- cal Debt Prioritization: State of the Art. A Systematic Literature Review, arXiv:1904.12538. ⇒ 338
  39. V. Lenarduzzi, F. Lomio, H. Huttunen, D. Taibi, Are SonarQube Rules Inducing Bugs?, IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2020, pp. 501-511. ⇒ 336, 339
  40. V. Lenarduzzi, V. Mandić, A. Katin, D. Taibi, How long do Junior Develop- ers take to Remove Technical Debt Items?, In Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Mea- surement(ESEM '20), 2020, Article 30, 1-6. ⇒ 336, 338
  41. A. Lerina, L. Nardi, Investigating on the Impact of Software Clones on Technical Debt, 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, pp. 108-112. ⇒ 349
  42. E. d. S. Maldonado, E. Shihab, Detecting and quantifying different types of self- admitted technical Debt, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 9-15. ⇒ 346
  43. M. A. Al Mamun, C. Berger, J. Hansson, Explicating, Understanding, and Man- aging Technical Debt from Self-Driving Miniature Car Projects, Sixth Interna- tional Workshop on Managing Technical Debt (MTD '14), 2014, pp. 11-18. ⇒ 346
  44. D. Marcilio, R. Bonifácio, E. Monteiro, E. Canedo, W. Luz, G. Pinto, Are Static Analysis Violations Really Fixed? A Closer Look at Realistic Usage of Sonar- Qube, IEEE/ACM 27th International Conference on Program Comprehension (ICPC), 2019, pp. 209-219. ⇒ 336, 339
  45. J. Y. Monteith, J. D. McGregor, Exploring software supply chains from a tech- nical debt perspective, 4th International Workshop on Managing Technical Debt (MTD '13), 2013, pp. 32-38. ⇒ 344
  46. A. Mori, G. Vale, M. Viggiato, J. Oliveira, E. Figueiredo, E. Cirilo, P. Jamshidi, C. Kastner, Evaluating Domain-Specific Metric Thresholds: An Empirical Study, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 41-50. ⇒ 348
  47. A. Nugroho, J. Visser, T. Kuipers, An empirical model of technical debt and interest, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, Waikiki, Honolulu, HI, USA, 1-8. ⇒ 343
  48. E. Parodi, S. Matalonga, D. Macchi, M. Solari, Comparing technical debt in stu- dent exercises using test driven development, test last and ad hoc programming, 2016 XLII Latin American Computing Conference (CLEI), 2016, pp. 1-10. ⇒ 336, 338
  49. I. Pigazzini, F. A. Fontana, V. Lenarduzzi, D. Taibi, Towards microservice smells detection, In Proceedings of the 3rd International Conference on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 92-97. ⇒ 350
  50. R. Plösch, J. Bräuer, M. Saft and C. Körner, Design Debt Prioritization: A De- sign Best Practice-Based Approach, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 95-104. ⇒ 348
  51. W. W. Royce, Managing the development of large software systems, Proceedings of IEEE WESCON, 1970, pp. 1-9. ⇒ 337
  52. N. Saarimäki, M. T. Baldassarre, V. Lenarduzzi and S. Romano, On the Ac- curacy of SonarQube Technical Debt Remediation Time, 45th Euromicro Con- ference on Software Engineering and Advanced Applications (SEAA), 2019, pp. 317-324. ⇒ 336, 338
  53. V. Lenarduzzi, N. Saarimäki, D. Taibi, On the Diffuseness of Code Technical Debt in Java Projects of the Apache Ecosystem, 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, pp. 98-107. ⇒ 349
  54. A. Shapochka, B. Omelayenko, Practical Technical Debt Discovery by Match- ing Patterns in Assessment Graph, 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), 2016, pp. 32-35. ⇒ 347
  55. V. Singh, W. Snipes, N. A. Kraft, A Framework for Estimating Interest on Tech- nical Debt by Monitoring Developer Activity Related to Code Comprehension, Sixth International Workshop on Managing Technical Debt (MTD '14), 2014, pp. 27-30. ⇒ 345
  56. A. Sundelin, J. Gonzalez-Huerta, K. Wnuk, The hidden cost of backward com- patibility: when deprecation turns into technical debt -an experience report, In Proceedings of the 3rd International Conference on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 67-76. ⇒ 350
  57. K. Szabados and A. Kovács, Technical debt of standardized test software, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 57-60. ⇒ 346
  58. R. Szalay, Á. Sinkovics, Z. Porkoláb, The Role of Implicit Conversions in Erro- neous Function Argument Swapping in C++, 20th International Working Con- ference on Source Code Analysis and Manipulation (SCAM), 2020, pp. 203-214. ⇒ 336
  59. J. Tan, D. Feitosa, P. Avgeriou, An empirical study on self-fixed technical debt, In Proceedings of the 3rd International Conference on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 11-20. ⇒ 350
  60. A. Tornhill, Prioritize Technical Debt in Large-Scale Systems Using CodeScene, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 59-60. ⇒ 348
  61. P. Wang, J. Yang, L. Tan, R. Kroeger, J. David Morgenthaler, Generating pre- cise dependencies for large software, 4th International Workshop on Managing Technical Debt (MTD '13), 2013, pp. 47-50. ⇒ 345
  62. N. Zazworka, C. Seaman, F. Shull, Prioritizing design debt investment opportu- nities, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, pp. 39-42. ⇒ 344
  63. * * * , MTD '11: Proceedings of the 2nd Workshop on Managing Technical Debt, 2011, ACM, New York, NY, USA. ⇒ 337, 338, 341, 343
  64. * * * , MTD '12: Proceedings of the Third International Workshop on Managing Technical Debt, 2012, IEEE Press ⇒ 337, 338, 341, 343, 344
  65. * * * , MTD '13: Proceedings of the 4th International Workshop on Managing Technical Debt, 2013, IEEE Press. ⇒ 337, 338, 341, 343, 344
  66. * * * , MTD '14: Proceedings of the 2014 Sixth International Workshop on Man- aging Technical Debt, 2014, IEEE Computer Society. ⇒ 337, 338, 341, 343, 345
  67. * * * , 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, IEEE Computer Society. ⇒ 337, 338, 341, 343, 346
  68. * * * , 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), 2016, IEEE Computer Society. ⇒ 337, 338, 341, 343, 347
  69. * * * , 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, IEEE Computer Society. ⇒ 337, 338, 341, 343, 347
  70. * * * , 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, IEEE Computer Society. ⇒ 337, 338, 341, 343, 348
  71. * * * , TechDebt '20: Proceedings of the 3rd International Conference on Technical Debt, 2020, ACM, New York, NY, USA. ⇒ 337, 338, 341, 343, 350
  72. * * * , 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, IEEE Computer Society. ⇒ 337, 338, 341, 343, 350