Generalized fixation invariant nuclei detection through domain adaptation based deep learning

Research output: Contribution to journalArticleScientificpeer-review

1 Downloads (Pure)


Nucleus detection is a fundamental task in histological image analysis and an important tool for many follow up analyses. It is known that sample preparation and scanning procedure of histological slides introduce a great amount of variability to the histological images and poses challenges for automated nucleus detection. Here, we studied the effect of histopathological sample fixation on the accuracy of a deep learning based nuclei detection model trained with hematoxylin and eosin stained images. We experimented with training data that includes three methods of fixation; PAXgene, formalin and frozen, and studied the detection accuracy results of various convolutional neural networks. Our results indicate that the variability introduced during sample preparation affects the generalization of a model and should be considered when building accurate and robust nuclei detection algorithms. Our dataset includes over 67 000 annotated nuclei locations from 16 patients and three different sample fixation types. The dataset provides excellent basis for building an accurate and robust nuclei detection model, and combined with unsupervised domain adaptation, the workflow allows generalization to images from unseen domains, including different tissues and images from different labs.

Original languageEnglish
Pages (from-to)1747-1757
JournalIEEE Journal of Biomedical and Health Informatics
Issue number5
Early online date2020
Publication statusPublished - 2021
Publication typeA1 Journal article-refereed


  • Deep learning
  • digital pathology
  • domain adaptation
  • formalinfixed
  • frozen section
  • nuclei detection
  • PAXgene-fixed
  • tissue fixation

Publication forum classification

  • Publication forum level 2

ASJC Scopus subject areas

  • Biotechnology
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Health Information Management


Dive into the research topics of 'Generalized fixation invariant nuclei detection through domain adaptation based deep learning'. Together they form a unique fingerprint.

Cite this