Moving with the times: predictive drift correction of multi-frame STEM
Authors
Matthew Mosse (1, 2), Jonathan JP Peters (1, 2, 3), Michele Conroy (4), Lewys Jones (1, 2, 3)
Affiliations
- School of Physics, Trinity College Dublin
- Advanced Microscopy Laboratory, Trinity College Dublin
- turboTEM Limited
- Department of Materials, London Centre of Nanotechnology, Imperial Henry Royce Institute, Imperial College London
Collecting many frames is often necessary in electron microscopy, for example in-situ dynamic imaging. But sample drift during a series acquisition may reduce the usable common area after alignment. Where significant drift is present, or during longer in-situ acquisitions, the feature of interest may even drift entirely out of the view. Here we use image correlation to diagnose and log this drift during acquisition. By predicting the stage drift for the next frame yet to be recorded, this foresight is used with a custom scan generator to scan over the same physical area every time.
For scanning transmission electron microscopy (STEM), image distortions are unavoidable during acquisitions, and can occur at different temporal and therefore spatial frequencies [1]. Collecting many, fast frames can reduce this effect, as lower frequency distortions become inter- rather than intra-frame effects and the frames can be re-aligned.
By fractionating dose over many fast frames, multi-frame imaging has also been shown to reduce the damage caused by the electron beam, giving time for thermal relaxation and charge dissipation [2], as well as allowing for the observation of dynamic processes such as phase changes.
Long multi-frame acquisitions can exhibit other issues including sample drift. This can cause the sample region imaged to change significantly such that after aligning and stacking the frames, a much smaller common area remains. This is inefficient both in terms of time and damage done to the sample, since much data has been discarded. For very long acquisitions the area of interest can drift out of the frame entirely.
We propose a solution by correcting this drift during the acquisition. By moving the area scanned by the STEM during an acquisition (relative to the column coordinate system), the region of the specimen scanned can be kept constant to compensate for the drift during the acquisition.
It is common to correct for the drift of the sample in light microscopy and life sciences [3], but those methods differ by correcting the position of the stage. This not as practical in electron microscopy, as the stage control is more challenging and may suffer from hysteresis, stick-slip motion or slow relaxation in rubber o-rings. Instead we adjust the scan area, which can be changed by single pixels or even sub-pixel, on the order of Angstroms.
The area scanned is normally a fraction of the instruments widest possible scan area. When the serial acquisition is started, and the two frames are captured, the relative translation of these frames is calculated using a correlation function such as cross correlation or mutual correlation [4]. The scanned area of the next frame can then be moved to compensate for this drift plus the extra drift expected from the first estimate. After just two frames, we can predict a linear drift rate for the third frame. After three frames are recorded the drift rates can be updated to yield refined estimates beyond just linear (e.g. linear velocity plus acceleration inference).
Figure 1: Process of correcting drift during acquisition. A scan area is chosen, and a frame is captured. Additional frames are then captured, and the amount of drift is calculated. The area scanned of the subsequent frame can be adjusted to compensate for this drift.
This process is similar to those presently used to correct stage drift, but those methods will always be at least one frame behind, only compensating for the drift of the previous frame. For large amounts of drift between frames, or for applications that require sub-pixel amounts of drift, this is not sufficient.
A function can be fitted to the recorded drift vectors and a prediction made for the next position. This addition can, ideally, reduce the drift of frames to below one pixel in every frame except for the second. The result of which can be seen in figure 2, where a stack of 25 frames has been stacked without additional alignment, and the effects of the drift can be almost eliminated.
Fig 2: (Left) drift of 1 pixel of drift in x and y is simulated for 25 frames, and the frames are summed. (Right) with same conditions, drift is calculated, predicted and compensated during a simulated acquisition, and frames were stacked.
For this approach to be pracically useful,the prediction must be computationally efficient enough to run live in parallel with the acquisition. For high signal-to-noise ratio images where the content of the image is expected to change (a phase change for example), a “live” mode can be used where the most recent images are used to estimate the velocity. For low SNR images, sets of frames might be stacked before correlation, improving computation speed.
This method should be possible to implement in most STEMs where programmable scan-generators are equipped. In addition, it should be generalisable to any scanning microscope, including SEM and AFM, the only requirement being the ability to change the scan area during multi-frame acquisitions. The code is written in python, allowing for maximum accessibility and interoperability.
This technique increases dose efficiency by reducing the area unnecessarily scanned and damaged. In addition, the operator’s time is saved is needed as smaller scan areas can be used and less time will be needed to allow the stage to settle. It can also produce sub-pixel drift over long time periods for adaptive scanning techniques, for which this is essential [5].
References
1] L. Jones, H. Yang, T. J. Pennycook, M. S. J. Marshall, S. Van Aert, N. D. Browning, M. R. Castell, and P. D. Nellist, Smart Align—a new tool for robust non-rigid registration of scanning microscope data, Adv. Struct. Chem. Imaging 1, 8 (2015).
[2] L. Jones, S. Wenner, M. Nord, P. H. Ninive, O. M. Løvvik, R. Holmestad, and P. D. Nellist, Optimising multi-frame ADF-STEM for high-precision atomic-resolution strain mapping, Ultramicroscopy 179, 57 (2017).
[3] R. McGorty, D. Kamiyama, and B. Huang, Active microscope stabilization in three dimensions using image correlation, Opt. Nanoscopy 2, 3 (2013).
[4] M. Van Heel, M. Schatz, and E. Orlova, Correlation functions revisited, Ultramicroscopy 46, 307 (1992).
[5] M. Mosse, J. J. P. Peters, B. W. Reed, D. J. Masiel, M. Conroy, and L. Jones, Event-Responsive Beam-Modulated STEM with Multi-Frame and Sparse Scanning, Microsc. Microanal. 30, (2024).
[6] MM, MC and LJ acknowledge funding from the SFI-EPSRC CDT-ACM (grants 18/EPSRC-CDT-3581 and EP/S023259/1), JJPP acknowledges funding from Research Ireland grant 19/FFP/6813, M.C. acknowledges funding from Royal Society Tata University Research Fellowship (URF∖R1∖201318), EPSRC NAME Programme Grant EP/V001914/1 & Royal Society Enhancement Award RF\ERE\210200EM1, LJ acknowledges funding from Research Ireland grant URF/RI/191637
Keywords
low-dose, STEM, multi-frame, drift correction