Homepage

mmc2021 Abstract Database

Multiple Object Tracking of Supported Nanoparticles during in situ Environmental TEM Studies of Nanocatalysts

Multiple Object Tracking of Supported Nanoparticles during in situ Environmental TEM Studies of Nanocatalysts

Session

Stream 2: EMAG - In-situ microscopy

Authors

Prof. Thierry Epicier (2, 4), Dr. Khuram Faraz (3, 4), Ass-Prof. Thomas Grenier (1), Prof. Christophe Ducottet (3)

Affiliations

1. CREATIS, Univ Lyon-INSA de Lyon-UCBL-CNRS-INSERM
2. IRCELYON, Univ Lyon-UCBL-CNRS
3. LaHC, Université Lyon, UJM-Saint Etienne-CNRS-Institut Optique Graduate School
4. MATEIS, Univ Lyon-INSA de Lyon-UCBL-CNRS

Keywords

Machine learning; Multiple Object Tracking; neural network; Environmental TEM; nanoparticles 

Abstract text

Characterizing at high spatial resolution nanoparticles (NPs) involved in heterogeneous catalysis is a key step of the understanding of their potential efficiency in catalytic processes. Using Environmental Transmission Electron Microscopy (ETEM), it is now possible to study these nanocatalysts under reactive conditions, i.e. under gas and temperature, thus mimicking or, in some best cases, reproducing experimental working conditions corresponding to their conditioning or activation. With such in situ or operando approaches, it is possible to investigate the reasons for deactivation of the catalyst, such as the growth of the NPs either by Ostwald ripening or coalescence [1]. Recording series of images at reasonable temporal frequencies such as permitted by modern fast cameras allows tracking the dynamic evolution of a quite large population of NPs exposed in situ to gas and heat stimuli. A meaningful and quantitative analysis of such observations can then be performed, assuming a correct control or possible damages induced by electron irradiation. To do so, tedious measurements are needed to detect the NPs and identify their trajectories when they possibly move and interact between them by diffusion processes. This is typically a multiple object tracking (MOT) problem, which is well-known and approached by automatic routines since a few decades for important societal subjects, such as pedestrians’ localization or traffic survey, see e.g. [2].

The purpose of the present work is to derive a robust and automatic pipeline to achieve this task on the basis of Machine Learning (ML) and MOT approaches. It will be applied to the calcination study of Pd NPs supported on delta-alumina, a well-known catalytic system for selective hydrogenation. Details of the TEM study were reported previously [3]. Basically, crushed samples were heated in situ in a Titan ETEM (FEI / Thermo Fisher Scientific Titan G2, 80-300 kV) up to 450°C in a few mbar of oxygen and ADF-STEM (Annular Dark Field Scanning TEM) sequences of images were acquired during up to 3 hours, with a variable time interval of a few minutes between successive frames.

For the detection of NPs in experimental images such as Figure 1a), we used the well-known Unet neural network [4]. Excellent results can be obtained once the network has been properly trained. Indeed, the key for a successful training is to rely on well-characterized and annotated data constituting the ground truth to which the network is confronted and which allows to train it. To avoid a tedious manual and subjective analysis of experimental data by one or several experts, we have developed a simple and efficient method based on simulated images for which all information is then known a priori.  According to the ideally incoherent nature of more or less high angle scattering processes controlling the collected intensity of ADF STEM images, it is easy to produce simulated images such as in Figure 1b). Any population of spherical (for simplicity) NPs, with known but varying chemical composition and size, can be generated on a supporting media with its own morphological characteristics (variable thickness, rugosity and pore distribution). We also developed a random walk routine to simulate dynamic sequences by moving NPs in agreement with expected size and intensity variations during coalescence, crossing or disappearance events. 

To fine-tune the network pre-optimized on such synthetized data, further simulated images are generated using true experimental micrographs as a support after having inpainted the inside of existing NPs (Figure 1c).

Figure 1: a) Low mag STEM image of the system Pd@d-Al₂O₃. b): Simulation exhibiting indicative similar features as compared to the experimental micrograph in a). c): Simulated image deduced from the experimental one (a) after erasing the NPs and generating new pseudo-circular ones. 

 

The second step of the approach concerns the NP tracking. We have implemented additional features to the algorithm developed by Milan et al. [2] initially dedicated to the tracking of pedestrian walks. It consists in a continuous energy minimization (CEM) which accounts for energy terms representing events such as illustrated in Figure 2. Regarding the nature of the problem treated here, we aimed at considering events and features that are mainly unrealistic in the case of humans or vehicles:

(i) fusion (coalescence) of two particles into a single one

(ii) less probable but still possible, of one particle which may divide into smaller ones

(iii) the required consistency of mass conservation during events (i) and (ii), as well as the global invariance of the volume of a single NP during its trajectory without interaction with others (hypothesis which can however be flawed if Ostwald ripening occurs involving too small particles or multimers and even single atoms which may have not been detected).

(iv) crossing of particles lying on upper and lower surfaces of the substrate.

Figure 2: Sketches of the tracking problematic. a): Positions of NPs identified on 13 frames of a dynamic sequence. b): Possible identification of 3 trajectories A, B, C. Note that the groups of positions B₂-B₄, B₈-B₁₀ and C₈-C₁₀ (C₉ supposed to be missed) show non-monotonous size variations which may indicate non-optimal tracking results. c): Better solution found according to a better identification of specific events: crossing in position B₃ (intensity increase due to the A₃+B₃ superimposition), split of A₆ into A’₇ (small then possibly missed) and A’’₇ and fusion A’₈+B₈. In these two last cases, addition of intensities and volume conservation can be discriminant. 

According to the above, size and intensity energies criteria were added to the CEM algorithm, which serve to identify and confirm fusion / division events such as sketched in figure 2.

Results of the Unet-based identification of NPs and of the modified CEM tracking approach will be illustrated on both simulated and experimental dynamic sequences [5].

References

[1] T.W. Hansen et al., Accounts of Chemical Research, 46 8 (2013) 1720.

[2] A. Milan et al., PAMI, 38 10 (2016) 2054.

[3] T. Epicier et al., Catalysis Today, 334, 15 (2019), 68.

[4] 0. Ronneberger et al., Lect. Notes Comput. Sci. 9351 (2015) 234.

[5] The authors thank the EUR SLEIGHT https://manutech-sleight.com/) for financial support, CLYM (www.clym.fr) for the access to the ETEM and IFPEN (Solaize, F) for providing samples.