When an Explanation is not Enough: An Overview of Evaluation Metrics of Explainable AI Systems in the Healthcare Domain

Despite the promising advantages in diagnostics and treatment that Artificial Intelligence (AI) and Machine Learning (ML) can bring to the healthcare domain, the complexity and black-box behavior of the AI/ML algorithms hinder the adoption by healthcare professionals and patients due to issues regarding explainability and trustworthiness of the results. Explainable AI (XAI) has emerged to support the need for understanding the AI/ML models' outputs and is expected to have a substantial relevance in the success of these models within the healthcare domain. Nevertheless, the information provided by XAI systems might be not enough to generate the required trustworthiness in the models. Thus, the existence of tools and metrics that allow domain experts and stakeholders to evaluate the explanations arises as needed solution. At the moment, there is an obvious lack of standardization and validation of metrics, and researchers require studies that compile the metrics together to know what, how, and why should be measured. This paper aims to provide an overview of the current metrics to evaluate XAI systems with a particular view on the healthcare domain. From the metrics identified and reviewed by following the PRISMA methodology, we present a taxonomy in which certain aspects are considered, such as the domain (general or healthcare) of the metric, as well as whether the expert is included in the validation process (human-in-the-loop). From our results, we observed many metrics developed in the general domain are being used for clinical XAI models. Nevertheless, it is essential to evaluate the XAI models in a more domain-specific manner, particularly because medical experts have valuable specialist information about the use cases that computer scientists might lack.