Invited Talk:    Enhancing Microscopy Image Analysis with In-Domain Pre-trained Encoders

Joshua Stuckner, NASA Glenn Research Center, USA

Abstract.

Establishing processing-structure-property relationships through microstructure quantification is a challenging yet crucial aspect of materials design and discovery. This work aims to demonstrate the efficacy of in-domain pre-training on microscopy tasks, using a large dataset of images called MicroNet. Over 30 Convolutional Neural Network (ConvNet) encoder architectures were pre-trained on over 100,000 labeled microscopy images from 54 material classes. These publicly released models can enhance the accuracy of any microscopy-related task that uses common ConvNet encoders. We applied these pre-trained encoders to downstream tasks including semantic segmentation, classification, regression, and instance segmentation. Notably, when trained with a single Ni-superalloy image, pre-training on MicroNet resulted in a 72.2% reduction in relative intersection over union error compared to ImageNet pre-training. This significant improvement demonstrates the potential of in-domain pre-training in generalizing better to out-of-distribution micrographs and achieving higher accuracy with less training data. We also discuss how these models informed the design of Ni-superalloys and welding parameters for Artemis rockets. Our findings suggest that transfer learning from large in-domain datasets can generate models with more useful learned feature representations for downstream tasks. Looking forward, we anticipate that larger public microscopy datasets could be leveraged to train even more effective encoders for microscopy tasks, particularly when using more data-hungry vision models and training techniques such as vision transformers and self-supervised learning.