Inpainting Approaches to Dose Control in High Resolution STEM and Ultrafast TEM

Session

EMAG - EM Data Processing & Analysis

Authors

Professor Nigel Browning (1)

Affiliations

1. University of Liverpool

Keywords

Compressed Sensing, Sub-Sampling, Inpainting, AI, STEM, RUEDI, Ultrafast

Abstract text

For many imaging and microanalysis experiments using state-of-the-art aberration corrected scanning transmission electron microscopes (STEM), the resolution and precision of the final result is primarily determined by the tolerance of the sample to the applied electron beam dose.   In the case of in-situ experiments, where the goal is to image a chemical or structural process as it evolves, the effect of the beam dose can be harder to unravel, as a change in structure/chemistry is the expectation of the experiment – when you are looking for a change in a complex experiment and do actually see one, it is natural to expect that the change you see is exactly the one you were looking for.  However, if the dose is not controlled, the kinetics of the observations can be dramatically changed by the beam, leading to a different structure/chemistry than would be expected from an ex-situ experiment under similar reaction conditions.  

Recent results at the University of Liverpool have shown that the optimal solution for dose control in any form of transmission electron microscopy (TEM) is to form the image from discrete locations of a small electron beam separated by as far as possible in space and time.  Instead of forming the image with an extended beam (as with TEM) or from a regular raster pattern (as in conventional STEM) this condition is satisfied ideally by moving the STEM probe over the area of the image using large jumps between the acquisition pixels.  This form of STEM imaging presents numerous challenges to the stability of the microscope, but these stability issues can be routinely overcome using either a form of random walk scanning, a calibrated random scanning or a mixture of conventional scanning and rapid beam blanking.  The larger than standard jumps between pixel acquisition locations in this methodology creates problems with image interpretation, as the gaps between locations of acquisition are missing information.   Fortunately, we can use Inpainting (a form of compressive sensing) to retrieve the missing information and retrieve the full image of the sample.  In this presentation, we will discuss the methodology behind the use of sub-sampling and Inpainting, with particular reference to the speed and efficiency of the reconstruction method and the potential for future real-time imaging.   In addition, the use of simulations to provide a starting point for image interpretation and the use of deep learning approaches to allow the microscope to adapt its own imaging conditions to the sample being studied, will also be discussed.  Finally, this complete integrated artificial intelligence work flow for image acquisition and processing ideally lends itself to integrated image analytics that can be used, for example, to quantify feature sizes and their rate of change directly from the sub-sampled datasets.

This approach to generating and interpreting low-dose images also has direct applications to ultrafast imaging, where the temporal resolution is very sensitive to the number of electrons used in pulses that form the image - all images are essentially low dose images with the electrons dispersed in space and time.  Currently, a new Relativistic Ultrafast Electron Diffraction and Imaging (RUEDI) national facility is working its way through the UKRI infrastructure development process.  The working design of RUEDI is based around a 4MeV RF source with a pulse duration of 10fs, coupled to both a diffraction and an imaging beam line (with sub-nm resolution). Should RUEDI be successful in its design goals, it will permit the direct observation of the atomic/electronic motions that control the fundamental (bio)-chemical interactions potentially behind transformative innovations in the science and technology of personalized medicine, energy storage, and clean growth. A key part of obtaining the combined spatial/temporal resolution goals for RUEDI is the use of  highest spatial resolution before/after static STEM images to create dictionaries that can be used to interpret the low-signal, ie poorer resolution, dynamic observations. In this presentation, the goals for RUEDI and how Inpainting methods will tie together spatial and temporal resolution will be discussed.