Machine learning-based test smell detection

Valeria Pontillo, Dario Amoroso d’Aragona, Fabiano Pecorelli, Dario Di Nucci, Filomena Ferrucci, Fabio Palomba

Research output: Contribution to journalArticleScientificpeer-review


Test smells are symptoms of sub-optimal design choices adopted when developing test cases. Previous studies have proved their harmfulness for test code maintainability and effectiveness. Therefore, researchers have been proposing automated, heuristic-based techniques to detect them. However, the performance of these detectors is still limited and dependent on tunable thresholds. We design and experiment with a novel test smell detection approach based on machine learning to detect four test smells. First, we develop the largest dataset of manually-validated test smells to enable experimentation. Afterward, we train six machine learners and assess their capabilities in within- and cross-project scenarios. Finally, we compare the ML-based approach with state-of-the-art heuristic-based techniques. The key findings of the study report a negative result. The performance of the machine learning-based detector is significantly better than heuristic-based techniques, but none of the learners able to overcome an average F-Measure of 51%. We further elaborate and discuss the reasons behind this negative result through a qualitative investigation into the current issues and challenges that prevent the appropriate detection of test smells, which allowed us to catalog the next steps that the research community may pursue to improve test smell detection techniques.

Original languageEnglish
Article number55
JournalEmpirical Software Engineering
Issue number2
Publication statusPublished - Mar 2024
Publication typeA1 Journal article-refereed


  • Empirical software engineering
  • Machine learning
  • Test code quality
  • Test smells

Publication forum classification

  • Publication forum level 3

ASJC Scopus subject areas

  • Software


Dive into the research topics of 'Machine learning-based test smell detection'. Together they form a unique fingerprint.

Cite this