Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction

Henrik Olsson; Kimmo Kartasalo; Nita Mulliqi; Marco Capuccini; Pekka Ruusuvuori; Hemamali Samaratunga; Brett Delahunt; Cecilia Lindskog; Emiel A.M. Janssen; Anders Blilie; ISUP Prostate Imagebase Expert Panel; Lars Egevad; Ola Spjuth; Martin Eklund

doi:10.1038/s41467-022-34945-8

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction

Henrik Olsson^*
, Kimmo Kartasalo
, Nita Mulliqi
, Marco Capuccini
, Pekka Ruusuvuori
, Hemamali Samaratunga
, Brett Delahunt
, Cecilia Lindskog
, Emiel A.M. Janssen
, Anders Blilie
, ISUP Prostate Imagebase Expert Panel
, Lars Egevad
, Ola Spjuth
, Martin Eklund

^*Corresponding author for this work

Research output: Contribution to journal › Article › Scientific › peer-review

91 Citations (Scopus)

70 Downloads (Pure)

Abstract

Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2%] without conformal prediction) while 175 (22%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (N = 49 for external scanner, N = 10 for external lab and scanner, and N = 12 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25%) without conformal prediction, while the system flags 143 (80%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.

Original language	English
Article number	7761
Number of pages	10
Journal	Nature Communications
Volume	13
Issue number	1
DOIs	https://doi.org/10.1038/s41467-022-34945-8
Publication status	Published - Dec 2022
Publication type	A1 Journal article-refereed

Funding

M.E. received funding from the Swedish Research Council (Vetenskapsrådet; 2020-00692), the Swedish Cancer Society (Cancerfonden; 21 1715 Pj), the Magnus Bergvall Foundation, Region Stockholm, Svenska Druidorden, Åke Wibergs Stiftelse, and Swedish e-Science Research Center (SeRC), Karolinska Institutet, and the Swedish Prostate Cancer Foundation (Prostatacancerförbundet). K.K. received funding from KAUTE Foundation, David and Astrid Hägelen Foundation, Oskar Huttunen Foundation, and Orion Research Foundation. Role of the funder: The funder had no role in the design of the study; data collection, analysis, and interpretation of the data; the writing of the manuscript; and the decision to submit the manuscript for publication. We want to thank Tony Ström, Carin Cavalli-Björkman, Astrid Björklund, and Britt-Marie Hune for assistance with scanning and database support.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Publication forum classification

Publication forum level 3

ASJC Scopus subject areas

General Chemistry
General Biochemistry,Genetics and Molecular Biology
General
General Physics and Astronomy

Access to Document

10.1038/s41467-022-34945-8Licence: CC BY

s41467-022-34945-8Final published version, 1.26 MBLicence: CC BY

https://urn.fi/URN:NBN:fi:tuni-202301021002Licence: CC BY

Cite this

Olsson, H., Kartasalo, K., Mulliqi, N., Capuccini, M., Ruusuvuori, P., Samaratunga, H., Delahunt, B., Lindskog, C., Janssen, E. A. M., Blilie, A., ISUP Prostate Imagebase Expert Panel, Egevad, L., Spjuth, O., & Eklund, M. (2022). Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction. Nature Communications, 13(1), Article 7761. https://doi.org/10.1038/s41467-022-34945-8

@article{f4d99a98ce8e42919765adea3295e541,

title = "Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction",

abstract = "Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1\%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2\%] without conformal prediction) while 175 (22\%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (N = 49 for external scanner, N = 10 for external lab and scanner, and N = 12 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2\%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25\%) without conformal prediction, while the system flags 143 (80\%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.",

author = "Henrik Olsson and Kimmo Kartasalo and Nita Mulliqi and Marco Capuccini and Pekka Ruusuvuori and Hemamali Samaratunga and Brett Delahunt and Cecilia Lindskog and Janssen, \{Emiel A.M.\} and Anders Blilie and \{ISUP Prostate Imagebase Expert Panel\} and Lars Egevad and Ola Spjuth and Martin Eklund",

note = "Funding Information: M.E. received funding from the Swedish Research Council (Vetenskapsr{\aa}det; 2020-00692), the Swedish Cancer Society (Cancerfonden; 21 1715 Pj), the Magnus Bergvall Foundation, Region Stockholm, Svenska Druidorden, {\AA}ke Wibergs Stiftelse, and Swedish e-Science Research Center (SeRC), Karolinska Institutet, and the Swedish Prostate Cancer Foundation (Prostatacancerf{\"o}rbundet). K.K. received funding from KAUTE Foundation, David and Astrid H{\"a}gelen Foundation, Oskar Huttunen Foundation, and Orion Research Foundation. Role of the funder: The funder had no role in the design of the study; data collection, analysis, and interpretation of the data; the writing of the manuscript; and the decision to submit the manuscript for publication. We want to thank Tony Str{\"o}m, Carin Cavalli-Bj{\"o}rkman, Astrid Bj{\"o}rklund, and Britt-Marie Hune for assistance with scanning and database support. Publisher Copyright: {\textcopyright} 2022, The Author(s).",

year = "2022",

month = dec,

doi = "10.1038/s41467-022-34945-8",

language = "English",

volume = "13",

journal = "Nature Communications",

issn = "2041-1723",

publisher = "Nature Research",

number = "1",

}

Olsson, H, Kartasalo, K, Mulliqi, N, Capuccini, M, Ruusuvuori, P, Samaratunga, H, Delahunt, B, Lindskog, C, Janssen, EAM, Blilie, A, ISUP Prostate Imagebase Expert Panel, Egevad, L, Spjuth, O & Eklund, M 2022, 'Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction', Nature Communications, vol. 13, no. 1, 7761. https://doi.org/10.1038/s41467-022-34945-8

TY - JOUR

T1 - Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction

AU - Olsson, Henrik

AU - Kartasalo, Kimmo

AU - Mulliqi, Nita

AU - Capuccini, Marco

AU - Ruusuvuori, Pekka

AU - Samaratunga, Hemamali

AU - Delahunt, Brett

AU - Lindskog, Cecilia

AU - Janssen, Emiel A.M.

AU - Blilie, Anders

AU - ISUP Prostate Imagebase Expert Panel

AU - Egevad, Lars

AU - Spjuth, Ola

AU - Eklund, Martin

N1 - Funding Information: M.E. received funding from the Swedish Research Council (Vetenskapsrådet; 2020-00692), the Swedish Cancer Society (Cancerfonden; 21 1715 Pj), the Magnus Bergvall Foundation, Region Stockholm, Svenska Druidorden, Åke Wibergs Stiftelse, and Swedish e-Science Research Center (SeRC), Karolinska Institutet, and the Swedish Prostate Cancer Foundation (Prostatacancerförbundet). K.K. received funding from KAUTE Foundation, David and Astrid Hägelen Foundation, Oskar Huttunen Foundation, and Orion Research Foundation. Role of the funder: The funder had no role in the design of the study; data collection, analysis, and interpretation of the data; the writing of the manuscript; and the decision to submit the manuscript for publication. We want to thank Tony Ström, Carin Cavalli-Björkman, Astrid Björklund, and Britt-Marie Hune for assistance with scanning and database support. Publisher Copyright: © 2022, The Author(s).

PY - 2022/12

Y1 - 2022/12

N2 - Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2%] without conformal prediction) while 175 (22%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (N = 49 for external scanner, N = 10 for external lab and scanner, and N = 12 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25%) without conformal prediction, while the system flags 143 (80%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.

AB - Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2%] without conformal prediction) while 175 (22%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (N = 49 for external scanner, N = 10 for external lab and scanner, and N = 12 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25%) without conformal prediction, while the system flags 143 (80%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.

U2 - 10.1038/s41467-022-34945-8

DO - 10.1038/s41467-022-34945-8

M3 - Article

C2 - 36522311

AN - SCOPUS:85144226226

SN - 2041-1723

VL - 13

JO - Nature Communications

JF - Nature Communications

IS - 1

M1 - 7761

ER -

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction

Abstract

Funding

UN SDGs

Publication forum classification

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this