Reproducibility in the technical debt domain
2022, Acta Universitatis Sapientiae, Informatica
https://doi.org/10.2478/AUSI-2021-00016Abstract
Context: It is crucial to understand how reproducible the measurement results in the scientific publications are, as reproducibility is one of the cornerstones of engineering. Objective: The goal of this study is to investigate the scientific publications presented at the premier technical debt conferences by understanding how reproducible the reported findings are. Method: We conducted a systematic literature review of 135 unique papers published at the "International Workshop on Managing Technical Debt" and the "International Conference on Managing Technical Debt", the premier scientific conference series on technical debt. Results: Only 44 of the investigated 135 papers presented numerical evidence and only 5 papers listed the tools, the availability of the tools, and the version of the tools used. For the rest of the papers additional information would have been needed for the potential reproducibility. One of the published papers even referred to a pornographic site as a source of a toolset for empirical research.
References (72)
- S. Ahmadjee, C. Mera-Gómez, R. Bahsoon, Assessing Smart Contracts Security Technical Debts, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 6-15. ⇒ 350
- A. Aldaeej, C. Seaman, From Lasagna to Spaghetti: A Decision Model to Man- age Defect Debt, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 67-71. ⇒ 348
- R. Alfayez, W. Alwehaibi, R. Winn, E. Venson, B. Boehm, A systematic liter- ature review of technical debt prioritization, In Proceedings of the 3rd Interna- tional Conference on Technical Debt (TechDebt '20), 2020, pp. 1-10. ⇒ 341
- R. R. Alfayez, P. Behnamghader, K. Srisopha, B. Boehm, An Exploratory Study on the Influence of Developers in Technical Debt, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 1-10. ⇒ 348
- S. Alhusain, Predicting Relative Thresholds for Object Oriented Metrics, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 55-63. ⇒ 351
- T. Amanatidis, N. Mittas, A. Chatzigeorgiou, A. Ampatzoglou, L. Angelis, The Developer's Dilemma: Factors Affecting the Decision to Repay Code Debt, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 62-66. ⇒ 348
- A. Ampatzoglou, A. Ampatzoglou, A. Chatzigeorgiou, P. Avgeriou, P. Abra- hamsson, A. Martini, U. Zdun, K. Systa, The Perception of Technical Debt in the Embedded Systems Domain: An Industrial Case Study, 2016 IEEE 8th In- ternational Workshop on Managing Technical Debt (MTD), 2016, pp. 9-16. ⇒ 347
- A. Ampatzoglou, A. Michailidis, C. Sarikyriakidis, A. Ampatzoglou, A. Chatzi- georgiou, P. Avgeriou, A Framework for Managing Interest in Technical Debt: An Industrial Validation, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 115-124. ⇒ 337, 347, 349, 352
- M. F. Aniche, G. A. Oliva, M. A. Gerosa, Are the Methods in Your Data Access Objects (DAOs) in the Right Place? A Preliminary Study, Sixth International Workshop on Managing Technical Debt (MTD '14), 2014, pp. 47-50. ⇒ 345
- P. C. Avgeriou, D. Taibi, A. Ampatzoglou, A. F. Fontana, T. Besker, A. Chatzi- georgiou, V. Lenarduzzi, A. Martini, A. Moschou, I. Pigazzini, N. Saarimäki, D. Sas, S. Toledo, A. Tsintzira. An Overview and Comparison of Technical Debt Measurement Tools, in IEEE Software, vol. 38, no. 3, pp. 61-71, May-June 2021. ⇒ 336, 338
- B. Barta, G. Manz, I. Siket, R. Ferenc, Challenges of SonarQube Plug-In Main- tenance, IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2019, pp. 574-578. ⇒ 336, 338
- T. Besker, A. Martini, J. Bosch, Technical debt cripples software developer pro- ductivity: a longitudinal study on developers' daily software development work, In Proceedings of the 2018 IEEE/ACM International Conference on Technical Debt (TechDebt '18), 2018, pp. 105-114. ⇒ 341
- J. Bohnet, J. Döllner, Monitoring code quality and development activity by software maps, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), ACM, 2011, New York, NY, USA, 9-16. ⇒ 341
- N. Brown, Y.g Cai, Y. Guo, R. Kazman, M. Kim, P. Kruchten, E. Lim, A. MacCormack, R. L. Nord, I. Ozkaya, R. Sangwan, C. Seaman, K. Sullivan, N. Zazworka, Managing technical debt in software-reliant systems, In Proceedings of the FSE/SDP workshop on Future of software engineering research (FoSER '10), 2010, pp. 47-52. ⇒ 337, 338, 341, 342, 343
- R. Capilla, T. Mikkonen, C. Carrillo, F. A. Fontana, I. Pigazzini, V. Lenarduzzi, Impact of Opportunistic Reuse Practices to Technical Debt, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 16-25. ⇒351
- A. Chatzigeorgiou, A. Ampatzoglou, A. Ampatzoglou, T. Amanatidis, Estimat- ing the breaking point for technical debt, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 53-56. ⇒ 346
- Carnegie Mellon University, Software Engineering Institute, Managing Technical Debt Workshops, previous editions, 2010, https://resources.sei.cmu.edu/news- events/events/techdebt/past.cfm, last visited: 2021.10.05. ⇒ 342, 345
- Y. Crespo, A. Gonzalez-Escribano, M. Piattini, Carrot and Stick approaches revisited when managing Technical Debt in an educational context, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 99-108. ⇒ 351
- D. Cruz, A. Santana, E. Figueiredo, Detecting bad smells with machine learning algorithms: an empirical study, In Proceedings of the 3rd International Confer- ence on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 31-40. ⇒ 350
- W. Cunningham, The WyCash Portfolio Management System, In Addendum to the proceedings on Object-oriented programming systems, languages, and appli- cations (Addendum) (OOPSLA '92). ACM, NY, USA, 1992, 29-30. ⇒ 337
- B. Curtis, J. Sappidi, A. Szynkarski, Estimating the size, cost, and types of Technical Debt, Third International Workshop on Managing Technical Debt (MTD'12), 2012, pp. 49-53. ⇒ 344
- R. R. de Almeida, R. do Nascimento Ribeiro, C. Treude, U. Kulesza, Business-Driven Technical Debt Prioritization: An Industrial Case Study, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 74-83. ⇒ 351
- M. A. de Freitas Farias, M. G. de Mendonça Neto, A. B. d. Silva, R. O. Spínola, A Contextualized Vocabulary Model for identifying technical debt on code com- ments, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 25-32. ⇒ 346
- J. de Groot, A. Nugroho, T. Bäck, J. Visser, What is the value of your software?, Third International Workshop on Managing Technical Debt (MTD'12), 2012, pp. 37-44. ⇒ 344
- M. di Biase, A. Rastogi, M. Bruntink, A. van Deursen, The Delta Maintain- ability Model: Measuring Maintainability of Fine-Grained Code Changes, 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, pp. 113-122. ⇒ 349
- D. Falessi, R. Kazman, Worst Smells and Their Worst Reasons, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 45-54. ⇒351
- D. Falessi, P. Kruchten, R. L. Nord, I. Ozkaya, Technical debt at the crossroads of research and practice: report on the fifth international workshop on managing technical debt, 2014, SIGSOFT Softw. Eng. Notes 39, 2, 31-33. ⇒ 343, 345
- D. Falessi, A. Reichel, Towards an open-source tool for measuring and visual- izing the interest of technical debt, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 1-8. ⇒ 346
- D. Falessi, A. Voegele, Validating and prioritizing quality rules for managing technical debt: An industrial case study, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 41-48. ⇒ 347
- F. A. Fontana, V. Ferme, S. Spinelli, Investigating the impact of code smells debt on quality code evaluation, Third International Workshop on Managing Technical Debt (MTD'12), 2012, pp. 15-22. ⇒ 344
- F. A. Fontana, V. Ferme, M. Zanoni, R. Roveda, Towards a prioritization of code debt: A code smell Intensity Index, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 16-24. ⇒ 346
- F. A. Fontana, R. Roveda, M. Zanoni, Technical Debt Indexes Provided by Tools: A Preliminary Discussion, 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), 2016, pp. 28-31. ⇒ 347
- I. Gat, J. D. Heintz, From assessment to reduction: how cutter consortium helps rein in millions of dollars in technical debt, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, Waikiki, Honolulu, HI, USA, 24-26. ⇒ 343
- R. Gomes, C. Siebra, G. Tonin, A. Cavalcanti, Fabio Q.B. da Silva, Andre L.M. Santos, R. Marques, An extraction method to collect data on defects and effort evolution in a constantly modified system, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, Waikiki, Honolulu, HI, USA, 27-30. ⇒ 343
- I. Griffith, D. Reimanis, C. Izurieta, Z. Codabux, A. Deo, B. Williams, The Cor- respondence Between Software Quality Models and Technical Debt Estimation Approaches, Sixth International Workshop on Managing Technical Debt (MTD '14), 2014, pp. 19-26. ⇒ 345
- I. Khomyakov, Z. Makhmutov, R. Mirgalimova, A. Sillitti, 2019, Automated Measurement of Technical Debt: A Systematic Literature Review, In Proceed- ings of the 21st International Conference on Enterprise Information Systems - Volume 2(ICEIS), 2019, pages 95-106. ⇒ 338
- J. R. Lahti, A. -P. Tuovinen, T. Mikkonen, Experiences on Managing Techni- cal Debt with Code Smells and AntiPatterns, 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, pp. 36-44. ⇒ 351
- V. Lenarduzzi, T. Besker, D. Taibi, A. Martini, F. A. Fontana, Techni- cal Debt Prioritization: State of the Art. A Systematic Literature Review, arXiv:1904.12538. ⇒ 338
- V. Lenarduzzi, F. Lomio, H. Huttunen, D. Taibi, Are SonarQube Rules Inducing Bugs?, IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2020, pp. 501-511. ⇒ 336, 339
- V. Lenarduzzi, V. Mandić, A. Katin, D. Taibi, How long do Junior Develop- ers take to Remove Technical Debt Items?, In Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Mea- surement(ESEM '20), 2020, Article 30, 1-6. ⇒ 336, 338
- A. Lerina, L. Nardi, Investigating on the Impact of Software Clones on Technical Debt, 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, pp. 108-112. ⇒ 349
- E. d. S. Maldonado, E. Shihab, Detecting and quantifying different types of self- admitted technical Debt, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 9-15. ⇒ 346
- M. A. Al Mamun, C. Berger, J. Hansson, Explicating, Understanding, and Man- aging Technical Debt from Self-Driving Miniature Car Projects, Sixth Interna- tional Workshop on Managing Technical Debt (MTD '14), 2014, pp. 11-18. ⇒ 346
- D. Marcilio, R. Bonifácio, E. Monteiro, E. Canedo, W. Luz, G. Pinto, Are Static Analysis Violations Really Fixed? A Closer Look at Realistic Usage of Sonar- Qube, IEEE/ACM 27th International Conference on Program Comprehension (ICPC), 2019, pp. 209-219. ⇒ 336, 339
- J. Y. Monteith, J. D. McGregor, Exploring software supply chains from a tech- nical debt perspective, 4th International Workshop on Managing Technical Debt (MTD '13), 2013, pp. 32-38. ⇒ 344
- A. Mori, G. Vale, M. Viggiato, J. Oliveira, E. Figueiredo, E. Cirilo, P. Jamshidi, C. Kastner, Evaluating Domain-Specific Metric Thresholds: An Empirical Study, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 41-50. ⇒ 348
- A. Nugroho, J. Visser, T. Kuipers, An empirical model of technical debt and interest, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, Waikiki, Honolulu, HI, USA, 1-8. ⇒ 343
- E. Parodi, S. Matalonga, D. Macchi, M. Solari, Comparing technical debt in stu- dent exercises using test driven development, test last and ad hoc programming, 2016 XLII Latin American Computing Conference (CLEI), 2016, pp. 1-10. ⇒ 336, 338
- I. Pigazzini, F. A. Fontana, V. Lenarduzzi, D. Taibi, Towards microservice smells detection, In Proceedings of the 3rd International Conference on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 92-97. ⇒ 350
- R. Plösch, J. Bräuer, M. Saft and C. Körner, Design Debt Prioritization: A De- sign Best Practice-Based Approach, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 95-104. ⇒ 348
- W. W. Royce, Managing the development of large software systems, Proceedings of IEEE WESCON, 1970, pp. 1-9. ⇒ 337
- N. Saarimäki, M. T. Baldassarre, V. Lenarduzzi and S. Romano, On the Ac- curacy of SonarQube Technical Debt Remediation Time, 45th Euromicro Con- ference on Software Engineering and Advanced Applications (SEAA), 2019, pp. 317-324. ⇒ 336, 338
- V. Lenarduzzi, N. Saarimäki, D. Taibi, On the Diffuseness of Code Technical Debt in Java Projects of the Apache Ecosystem, 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, pp. 98-107. ⇒ 349
- A. Shapochka, B. Omelayenko, Practical Technical Debt Discovery by Match- ing Patterns in Assessment Graph, 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), 2016, pp. 32-35. ⇒ 347
- V. Singh, W. Snipes, N. A. Kraft, A Framework for Estimating Interest on Tech- nical Debt by Monitoring Developer Activity Related to Code Comprehension, Sixth International Workshop on Managing Technical Debt (MTD '14), 2014, pp. 27-30. ⇒ 345
- A. Sundelin, J. Gonzalez-Huerta, K. Wnuk, The hidden cost of backward com- patibility: when deprecation turns into technical debt -an experience report, In Proceedings of the 3rd International Conference on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 67-76. ⇒ 350
- K. Szabados and A. Kovács, Technical debt of standardized test software, 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, pp. 57-60. ⇒ 346
- R. Szalay, Á. Sinkovics, Z. Porkoláb, The Role of Implicit Conversions in Erro- neous Function Argument Swapping in C++, 20th International Working Con- ference on Source Code Analysis and Manipulation (SCAM), 2020, pp. 203-214. ⇒ 336
- J. Tan, D. Feitosa, P. Avgeriou, An empirical study on self-fixed technical debt, In Proceedings of the 3rd International Conference on Technical Debt (TechDebt '20), 2020, ACM, New York, NY, USA, 11-20. ⇒ 350
- A. Tornhill, Prioritize Technical Debt in Large-Scale Systems Using CodeScene, 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, pp. 59-60. ⇒ 348
- P. Wang, J. Yang, L. Tan, R. Kroeger, J. David Morgenthaler, Generating pre- cise dependencies for large software, 4th International Workshop on Managing Technical Debt (MTD '13), 2013, pp. 47-50. ⇒ 345
- N. Zazworka, C. Seaman, F. Shull, Prioritizing design debt investment opportu- nities, In Proceedings of the 2nd Workshop on Managing Technical Debt (MTD '11), 2011, pp. 39-42. ⇒ 344
- * * * , MTD '11: Proceedings of the 2nd Workshop on Managing Technical Debt, 2011, ACM, New York, NY, USA. ⇒ 337, 338, 341, 343
- * * * , MTD '12: Proceedings of the Third International Workshop on Managing Technical Debt, 2012, IEEE Press ⇒ 337, 338, 341, 343, 344
- * * * , MTD '13: Proceedings of the 4th International Workshop on Managing Technical Debt, 2013, IEEE Press. ⇒ 337, 338, 341, 343, 344
- * * * , MTD '14: Proceedings of the 2014 Sixth International Workshop on Man- aging Technical Debt, 2014, IEEE Computer Society. ⇒ 337, 338, 341, 343, 345
- * * * , 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), 2015, IEEE Computer Society. ⇒ 337, 338, 341, 343, 346
- * * * , 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), 2016, IEEE Computer Society. ⇒ 337, 338, 341, 343, 347
- * * * , 2018 IEEE/ACM International Conference on Technical Debt (TechDebt), 2018, IEEE Computer Society. ⇒ 337, 338, 341, 343, 347
- * * * , 2019 IEEE/ACM International Conference on Technical Debt (TechDebt), 2019, IEEE Computer Society. ⇒ 337, 338, 341, 343, 348
- * * * , TechDebt '20: Proceedings of the 3rd International Conference on Technical Debt, 2020, ACM, New York, NY, USA. ⇒ 337, 338, 341, 343, 350
- * * * , 2021 IEEE/ACM International Conference on Technical Debt (TechDebt), 2021, IEEE Computer Society. ⇒ 337, 338, 341, 343, 350