Removing barriers to advanced imaging and machine learning-based analysis

Brackenbury, Cimini, Lacy, Liu, Malcolm, O’Toole, Papaleo, Powell, Wiggins

March 2025

Abstract

Over the past decade, imaging and image analysis technologies have advanced significantly, but not everyone around the world has benefited equally. Some research groups are trying to make high-quality microscope hardware and software more affordable, but there is still a lot to do. Brightfield microscopy, a common technique, is used in high and low resource settings throughout the world. Recent improvements in hardware and software have made it possible to get more information from these microscopes.

In our project, we plan to use our expertise in developing new imaging methods and machine learning to create new hardware and combined software. We will focus on quantitative phase imaging (ptychography) and brightfield microscopy, especially for time-lapse imaging of live cells. We will work closely with partners in low- and middle-income countries (LMICs) to develop methods that allow researchers to get more information from images using low cost equipment.

We will set up two ptychography hubs in South America and Africa to provide access to equipment and collaborate on developing a low-cost imaging and analysis system. By improving and sharing brightfield microscopy and ptychography hardware and deep learning algorithms, we aim to speed up cell biology and imaging research worldwide.

CellPhe & the Cell Atlas

In previous work Wiggins et al. have developed a pipeline for image analysis, referred to as CellPhe. It involves the extraction of a large number of image features that can be identified with particular cell characteristics. The features are then assessed and/or refined in the context of classification problems brought by the pipeline user.

CellPhe helps scientists and clinicians to find relationships between cell behaviours and cell features. Importantly,  the physical interpretation of the features is preserved. The analysis is exploratory in nature and the computation is human-interpretable.

The current project involves scaling up CellPhe and demonstrating its applicability to a wider range of cell types and image modalities. In doing so we create a library of images and feature data, constituting our Cell Atlas.

Deep-learning approaches to bio-imaging

The current project involves the testing and development of neural network models for feature extraction and image processing. Some aspects of the human-interpretability that characterized CellPhe are forfeited in exchange for improved predictive ability.

Priority applications include the production of software tools for

• Approximate translation between image types, with principle application to pytchography and brightfield images,

• Recovering and enhancing distorted/corrupted/noisy images,

• Extracting meaningful image features without costly intermediate computation (e.g. image segmentation, cell tracking).

Architectures and training strategies

Architectures: ‘Network architecture’ refers to the organization and scheduling of calculations which, when strung together, constitute a function whose input is observed data and whose output is actionable information. In the imaging domain several classes of architecture have gained popularity. These include VAEs, GANS and U-nets. Our team is working in parallel with each of these to test their relevance to the problems we are most interested in - namely, robustness, versatility, ease of use and scalable computational demand. All the architectures are characterized by form of 'computational bottleneck' that forces them to reduce images down to a small number of 'features' that retain as much useful information as possible.

Training strategies: Our deep-learning models use the parameters and data available to them to optimize objective functions. These objective functions are provided by us and require a significant amount of care to specify appropriately. So called ‘adversarial’ models, for example, are incentivized to trick people or other models into thinking an artificial image is real. ‘Autoencoders’ are incentivized to find low-dimensional representations of images from which they can be reconstructed with minimal information loss. We are working on the specification of appropriate objective functions to encourage them to learn useful things. The functions also constitute metrics for our models that allow us to select the most promising ones.

The diagram below represents a kind of schematic that is shared by most, if not all, of our models. They decompose the information in an image into a relatively small set of informative 'features' and a much larger set of 'details'. The model learns what is a feature and what is a detail according to whether it is useful to identify cell-types and/or to reconstruct the original image. Sometimes we apply additional transformations to the features in order to modify the image. For example, we may want to enhance important features or re-balance them in a way that changes the image style. The dashed line originating from the 'details' box indicates that it is sometimes useful to discard or suppress them when producing an output image.

Challenges and targets

Priorities for the coming months include:

• Designing software whose computational demand can scale to the resources available,

• Improving data processing pipelines to accelerate the process of model development,

• Stress-testing models with varied and extreme image/noise features,

• Logged and commented experimentation with novel model architectures and objective functions,

• Understanding the trade-offs between feature interpretability and predictive ability.

Experiments and preliminary findings

Preliminary results include

• The production of >2TB of image data from a wide range of cell types and image modalities,

• First contributions to the Cell Atlas of image/feature types,

• Prototype VAE, GAN and U-net neural network models for image-to-image translation and feature extraction.

Preliminary results: image enhancement

The images below represent preliminary results from one of our u-net models. They are liable to be updated as more progress is made. The u-net tries to distill from image data information that is useful for classifying the cells. More specifically, it looks for features that are invariant to the imaging modality so that it is not reliant on any particular type of microscope or camera. 

The feature vectors are just sequences of numbers that are useful for down-stream tasks such as clustering or classification. To gain some insight into the correspondence between feature vectors and properties of the image we can reconstruct the images, but with amplified versions of the original feature vectors. The pairs of images below are examples of original and feature-amplified images. It is one of our working hypotheses that these images will be useful for clinicians who want to know what the u-net is looking at in a picture when it is deciding how to classify it.

Preliminary results: image translation

The images below, which again represent preliminary results with a prototype model, show the output of a u-net that has been trained to guess the content of a phase-constrast image given a corresponding bright-field image. This training is possible due to the large quantity of image pairs that we have collected over the past few months. We observe that, to a great extent, the model is able to see features in the bright-field images necessary for inferring the phase-contrast image.

Team members

The team consists of Will Brackenbury, Beth Cimini, Stuart Lacy, Le Liu, Jodie Malcolm, Peter O'Toole, Andréa Papaleo, Ben Powell and Laura Wiggins. We bring together expertise from cell biology, microscopy, computer science and mathematics.