Artificial intelligence (AI)-powered algorithms are now influencing many aspects of our day-to-day life, from providing movies/music recommendations to controlling self-driving cars. These algorithms are also increasingly used in the lab to aid biomedical research. In particular, the ability to analyse and process images using AI is slowly revolutionizing the quality and quantity of data we collect from microscopy images. In fact, AI-based algorithms can now be applied to perform virtually any high-performance image analysis tasks such as classifying images, detecting and segmenting objects, aligning images or improving image quality by removing noise or increasing image resolution. This short feature article briefly underlies the principles behind using AI algorithms to analyse microscopy images with a specific focus on segmentation and denoising.
Over the last 400 years, microscopes have allowed us to observe objects too small to be seen with the naked eye. Today, light and electron microscopies are leading technologies used worldwide to perform research and diagnostics. A recent survey indicated that up to 90% of life sciences’ publications utilize microscopy in one way or another. Modern microscopes are typically connected to digital cameras and a microscopy session often leads to the acquisition of hundreds to thousands of images. Like any digital image, microscopy images are composed of pixels arranged in a two-dimensional grid, each with a finite, discrete numerical value. Microscopy images are rich in information and, to gain meaningful results, images need to be processed and analysed using computers. Because of this, image analysis software are continuously developed to support life science research. Over the last 5 years, software developers have turned to artificial intelligence (AI) to revolutionize microscopy image analyses. In particular, deep learning (DL), a subset of AI capable of learning in an unsupervised manner, often outperforms conventional image processing strategies. For instance, DL is increasingly employed for image analysis tasks such as detecting objects or improving image quality by enhancing resolution or removing unwanted noise.
Deep learning, how does it work?
One of the reasons DL is now outperforming other image analysis strategies is that when using DL, the data always comes first. DL-based image analysis usually requires using algorithms called artificial neural networks (ANNs). One key feature of these algorithms is that they learn how to perform a task (for instance, removing noise) using example data. Therefore, before using an ANN to process images, it first needs to be trained. During the training process, the ANN is presented with a large set of representative images and their corresponding expected results. For instance, in the case of denoising, the ANN learns how to remove noise by comparing paired high-quality and noisy images provided by the user. With this training dataset, the ANN builds a model of the transformation that needs to be applied to the images to obtain the desired result (e.g., removing noise). Once this model has been generated, it can be applied to analyse new images. Therefore, using DL is a two-step process involving first training a DL model and then using this model to analyse images. The training is the most challenging part of the process as it requires specialized knowledge, access to high-quality training datasets and significant computing power (typically powerful graphics processing units [GPUs]). In contrast, using trained DL models can be relatively straightforward and rarely requires access to powerful workstations or servers. In fact, various trained models capable of processing microscopy images are readily available on websites or mobile devices (e.g., https://www.cellpose.org/ or http://www.nucleaizer.org/).
While the use of DL for microscopy is currently exploding, the underlying technology is not new. The first small but functional ANNs were described in the mid-1960s. However, due to the computing power required to train them, these algorithms became practical only after GPUs became widely available, in large part due to the booming gaming industry in the mid-2000s. The release of AlexNet (2012), an algorithm capable of classifying images with great accuracy, was then the most influential demonstration that ANNs can be powerful algorithms for image processing. Several algorithms then demonstrated the enormous potential of DL to analyse microscopy images, including U-Net for image segmentation (2015) and CSBDeep CARE to restore images (2018).
Deep learning to identify and segment objects in microscopy images
One common, yet complex, image analysis task is to detect and identify specific objects present in microscopy images. In computer vision, segmentation separates objects of interest from the background, while instance segmentation separates these objects from the background and each other. This can be used, for instance, for detecting cells, nuclei or other organelles from microscopy images. Once segmented, objects can then be counted and their shapes (e.g., perimeter, area, aspect ratio) and properties (e.g., intensity, texture) measured.
Manual segmentation is time-consuming, requires expert knowledge and is often a bottleneck when analysing large datasets. Therefore training computer algorithms to do the work automatically is a very attractive process. DL algorithms have proven powerful tools to aid with segmentation tasks. Indeed they combine expert-level performance with high-throughput analysis. Popular segmentation algorithms include U-Net, StarDist (segmentation of nuclei) or CellPose (cell segmentation). These algorithms typically use a U-Net-based architecture that aims to classify each pixel within an input image as background or as an object of interest by learning from large amounts of annotated data.
Therefore, to use such algorithms, users need first to provide examples of images and matching results to teach the DL algorithm to segment their data. The generation of such training datasets can be quite cumbersome as it often involves manually annotating 30–100 images before having enough data to train a DL model. In some contexts, the production of suitable training datasets is so time-consuming that citizen science projects have been created. For instance, with the ‘Etch a Cell’ project (https://etchacell.crick.ac.uk/), anyone can help scientists segment the nuclear envelope or find mitochondria from electron microscopy images.
For regular segmentation tasks, such as cell or nuclei segmentation, one powerful approach is to produce general models with high reusability potential using an extensive and diverse training dataset. For example, popular nuclei (StarDist, nucleAIzer, CellPose) or cell segmentation models (CellPose) have been released (Figure 1). When possible, this approach greatly facilitates the adoption of DL for image segmentation as users can directly test these models on their images.
Example illustrating how DL can be used to detect cancer cells from microscopy images. Left picture: Original microscopy image. Right picture: Image where each detected cancer cell has a different colour. In this case, the CellPose segmentation algorithm was used via the ZeroCostDL4Mic platform. The original images are of breast cancer cells labelled with SiR-DNA (magenta) to visualize their nuclei and with lifeact-RFP (green) to visualize their actin cytoskeleton. Images were acquired using a spinning disk confocal microscope.
Example illustrating how DL can be used to detect cancer cells from microscopy images. Left picture: Original microscopy image. Right picture: Image where each detected cancer cell has a different colour. In this case, the CellPose segmentation algorithm was used via the ZeroCostDL4Mic platform. The original images are of breast cancer cells labelled with SiR-DNA (magenta) to visualize their nuclei and with lifeact-RFP (green) to visualize their actin cytoskeleton. Images were acquired using a spinning disk confocal microscope.
Deep learning to restore and improve microscopy images
Fluorescence microscopy is a common microscopy strategy utilized in life science research. By labelling specific molecules (e.g., proteins) with fluorescent tags, biologists can use lasers to observe cellular components in samples selectively. However, due to the nature of light and the imaging system used, images acquired using fluorescence microscopy have an intrinsic resolution limit (the amount of detail that can be resolved) and are prone to noise. DL has shown great potential to address both of these challenges.
The amount of light collected when performing fluorescence microscopy experiments is defined by the amount and nature of the fluorescent molecules observed as well as by the laser intensities used during the imaging session. As high laser intensities damage biological samples, it is crucial to apply low laser intensities when imaging live samples, which results in noisy images. Therefore denoising strategies can be beneficial to improve microscopy image quality. Because of this, several DL-based denoising tools have been developed (Figure 2). These include popular open-source tools (e.g., CSBDeep CARE, Noise2Void, 3D RCAN) and commercial software (e.g., Denoise.AI, the Apeer platform or Aivia). As described earlier, most DL-based denoising methods require corresponding noisy input images and their noise-free counterparts to train DL models. As denoising models tend to be specific to a particular experiment, this places a burden on the users who need to generate dedicated training datasets to restore their images. However, innovative strategies such as Noise2Void now also allow training DL models directly from noisy images by predicting the intensity of every pixel using the surrounding pixels. This can considerably simplify the use of DL methods to denoise images.
Example illustrating how DL can be used to remove noise from microscopy images. Left picture: Original microscopy image. Right picture: Denoised image. In this case, the CSBDeep CARE algorithm was used via the ZeroCostDL4Mic platform. The images displayed are breast cancer cells labelled with SiR-DNA, to visualize nuclei, taken using a spinning disk confocal microscope. A region of interest (ROI) is magnified.
Example illustrating how DL can be used to remove noise from microscopy images. Left picture: Original microscopy image. Right picture: Denoised image. In this case, the CSBDeep CARE algorithm was used via the ZeroCostDL4Mic platform. The images displayed are breast cancer cells labelled with SiR-DNA, to visualize nuclei, taken using a spinning disk confocal microscope. A region of interest (ROI) is magnified.
One of the limitations of light microscopy is that the diffraction of light precludes the visualization of details below ∼200 nm. Over the last two decades, multiple methods have been developed to bypass this limit. Due to the interest in improving the resolution of microscopy images, DL-based strategies have also been developed. For instance, algorithms such as PSSR or DFCAN can increase the number of pixels in an image and predict missing details. Other DL algorithms can aid the post-processing analysis of images that are required for most super-resolution techniques. This is the case, for instance, for DeepSTORM or DECODE, two algorithms dedicated to the reconstruction of single-molecule microscopy data.
The pitfall of using deep learning
The ability to learn how to perform an analysis from example data is both the principal strength and the main weakness of DL. By learning directly from the data, the algorithms identify the most suitable way to perform the analysis, leading to excellent performances. However, trained DL models are only as good as the data used to train them. Suppose the training dataset is inadequate due to bias (e.g., images acquired using only one instrument) or simply too small (not enough data to learn from). In that case, the resulting model may produce unwanted results (e.g., poor performance and artefacts) when new images are analysed. This is especially the case if the images to process are too different from those used during the training. Because of this, a high-quality and diverse training dataset is crucial when using DL, something which can be very difficult to generate. Because of these issues, researchers are developing tools to increase the size of training datasets artificially, a process classically referred to as data augmentation, via various image manipulation (rotations, mirroring, adding noise, shear, etc.).
Getting started with deep learning for microscopy
As DL is revolutionizing the way we analyse images, you may wonder how you could get started with using DL at home. The vast majority of DL-based software are distributed as Python packages and therefore require the ability to code in Python. Often the easiest way to test DL tools with your images is to test the models provided as web interfaces (e.g., https://www.cellpose.org/ or http://www.nucleaizer.org/). However, these platforms remain limited and do not typically allow users to train their own models. More flexible tools that do not require programming skills are also becoming available; these include, for instance, plugins for Fiji/ImageJ (e.g., DeepImageJ or CSBDeep), ImJoy, DeepMIB, ilastik or Cell Profiler.
In collaboration with Dr Ricardo Henriques (Instituto Gulbenkian de Ciência, PT) and several other laboratories, we recently developed ZeroCostDL4Mic. ZeroCostDL4Mic is an entry-level teaching and deployment DL platform for microscopy images. In practice, ZeroCostDL4Mic is a collection of Python notebooks featuring a simple graphical interface. ZeroCostDL4Mic takes advantage of Google Colab, a free platform that allows running Python code on Google’s cloud servers and provides the computational resources needed to train and deploy DL networks. While the underlying code is hidden by default, it remains easily accessible, allowing users to explore and modify it at their leisure. Importantly, ZeroCostDL4Mic is freely available online and is fully open access (https://github.com/HenriquesLab/ZeroCostDL4Mic).
Outlook
DL now allows researchers to perform increasingly complex image analysis tasks, and the future will surely bring more powerful methods. DL algorithms are also being incorporated as part of the software controlling microscopes. In the future, this will enable self-driving microscopes. One advantage will be the design of reactive microscopy experiments. For instance, the microscopy software can learn to recognize specific events and trigger a particular response, such as moving the stage to follow cells over time or using different acquisition parameters.
Despite its incredible potential, using DL appropriately can remain challenging and requires skill and know-how not yet typically taught during biology studies. Therefore one challenge ahead is to educate the next generation of microscopists to adequately and responsibly utilize these powerful technologies. Another challenge is avoiding falling into the AI hype and remembering that other techniques may be more appropriate, more robust and sometimes quicker to analyse images.■
Further reading
von Chamier, L., Laine, R.F. and Henriques, R. (2019) Artificial intelligence for microscopy: what you should know. Biochem. Soc. Trans. 47, 1029–1040. DOI: 10.1042/BST20180391
This review describes the various use of artificial intelligence for microscopy.
Belthangady, C. and Royer, L.A. (2019) Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods16, 1215–1225. DOI: 10.1038/s41592-019-0458-z
Another review describing the use of artificial intelligence for microscopy.
Lucas, A.M., Ryder, P.V., Li, B., et al. (2021) Open-source deep-learning software for bioimage segmentation. Mol. Biol. Cell32, 823–829.DOI: 10.1091/mbc.E20-10-0660
This review describes several open-source tools that can be used to segment microscopy images using deep learning.
Ronneberger, O., Fischer, P. and Brox, T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv 1505.04597. DOI: abs/1505.04597
This paper describes the U-Net architecture and how it can be used to analyse biomedical images.
von Chamier, L., Laine, R.F., Jukkala, J. et al. (2021) Democratising deep learning for microscopy with ZeroCostDL4Mic. Nat. Commun.12, 2276. DOI: 10.1038/s41467-021-22518-0
This paper describes the ZeroCostDL4Mic platform.
Weigert, M., Schmidt, U., Boothe, T. et al. (2018) Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods15, 1090–1097. DOI: 10.1038/s41592-018-0216-7
This paper describes the CSBDeep CARE algorithm that can be used to restore microscopy images using deep learning.
Stringer, C., Wang, T., Michaelos, M. et al. (2021) Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods18, 100–106. DOI: 10.1038/s41592-020-01018-x
This paper describes the CellPose algorithm that can be used to segment cell microscopy images using deep learning.
Author information
Guillaume Jacquemet is a group leader at Åbo Akademi University (Turku, Finland). His laboratory uses advanced microscopy techniques to study how cancer cells interact with their environment during cancer cell metastasis. He is also the co-developer of the ZeroCostDL4Mic platform that aims at making deep learning for microscopy more accessible. Email: [email protected]