Deep Fourier channel attention networks precisely super-resolves the densely crisscrossing F-actin cytoskeleton from diffraction-limited raw SIM images acquired from COS-7 cell.

(Image by Dr. LI Dong's group)

LI Dong's group from Institute of Biophysics, Chinese Academy of Sciences, and Bioland Laboratory, Guangzhou in collaboration with DAI Qionghai's group from Department of Automation, Tsinghua University developed the deep Fourier channel attention network (DFCAN) and its derivative that was trained with generative adversarial network (GAN) strategy, termed as DFGAN. DFCAN and DFGAN are able to outperform the conventional super-resolution (SR) microscopy under most of routine live-cell SR imaging conditions, thus hold a great potential to democratize SR imaging with commonly used diffraction-limited microscope. The related study was published in Nature Methods on January 21, 2021, entitled "Evaluation and development of deep neural networks for image super-resolution in optical microscopy".

In this work, the authors home-built the multi-modality structured illumination microscope (Multi-SIM) that first integrated their previously developed SIM techniques, including TIRF-SIM, Nonlinear SIM [Science 349, 944, 2015], and Grazing Incidence (GI-SIM) [Cell 175, 1430-1442. 2018] together. Multi-SIM allowed acquire high quality experimental dataset of well-registered pairs of diffraction-limited wide-field and ground truth SIM images. This dataset named as "BioSR" now is open access for all researchers.

The comprehensive dataset provides a benchmark to systematically assess the fidelity and quantifiability of various deep learning super-resolution (DLSR) models in terms of the complexity of observed biological structures, the signal-to-noise ratio (SNR) of input low-resolution images, and the desired upscaling-factors. However, it turns out that current DLSR models hardly gain as high-fidelity SR information as the conventional hardware SR microscopy under the commonly used live-cell imaging conditions, which potentially impedes their widespread applications onto practical experiments.

The schematic of Fourier channel attention mechanism and two-color live-cell imaging via DFCAN/DFGAN-SIM of mitochondria and endoplasmic reticulum in a COS-7 cell.

(Image by Dr. LI Dong's group)

To further improve the DLSR imaging performance, the joint research group devised the deep Fourier channel attention network (DFCAN) and deep Fourier generative adversarial network (DFGAN) that utilized the power-spectrum-coverage (PSC) of individual feature maps to adaptively rescale their weightings when propagating through the network. Because PSC difference in Fourier domain is more prominent than the difference of detailed structures in spatial domain between diffraction-limited input images and ground truth SR images, the discriminative learning ability of DFCAN/DFGAN is significantly enhanced. Therefore, DFCAN/DFGAN is able to outperform the conventional SR microscopy under most of routine live-cell SR imaging conditions, suggesting that it has great potential to democratize super-resolution imaging with commonly used conventional microscope.

To demonstrate the capability of DFCAN and DFGAN for SR live-cell imaging, the authors applied DFCAN/DFGAN to the study of fragile biological processes that are challenging for conventional SR microscopy and other DLSR models, revealing:

· The partition and merging of mitochondrial nucleoids are regulated by the remodeling of mitochondrial inner membrane ultrastructure.

· Not only the large plant cells, but also relatively small animal cells are also capable of generating spiral rotational streaming to facilitate the circulation of intracellular contents, so as to maintain homeostasis of intracellular environment.

· Mitochondrial division and coalescing loci often locate at the membrane contact sites between ER and mitochondria.

· Actin filaments interact with clathrin coated pits in the late stage of endocytosis that facilitates the pits detaching from plasma membrane.

"We believe the assessment platform and the concept of DFCAN/DFGAN developed in this work fill unmet need for evaluating the performance of deep learning imaging methods, and shed new light on future development of SR microscopy" said Prof. LI Dong from Institute of Biophysics.

Paper link: https://www.nature.com/articles/s41592-020-01048-5

Contact: LI Dong

Institute of Biophysics, Chinese Academy of Sciences

Beijing 100101, China

Email: lidong@ibp.ac.cn

(Reported by Dr. LI Dong's group)