US 12,272,037 B2
Dual-mode restoration microscopy
Li-Ling Yang, Berlin (DE); Christian Conrad, Berlin (DE); Foo Wei Ten, Berlin (DE); and Roland Eils, Berlin (DE)
Appl. No. 17/800,225
Filed by Charité-Universitätsmedizin Berlin, Berlin (DE)
PCT Filed Feb. 17, 2021, PCT No. PCT/EP2021/053904
§ 371(c)(1), (2) Date Aug. 17, 2022,
PCT Pub. No. WO2021/165330, PCT Pub. Date Aug. 26, 2021.
Claims priority of application No. 20157777 (EP), filed on Feb. 17, 2020.
Prior Publication US 2023/0069794 A1, Mar. 2, 2023
Int. Cl. G06T 5/00 (2024.01); G02B 21/06 (2006.01); G02B 21/36 (2006.01); G06T 5/50 (2006.01); G06T 5/80 (2024.01); G06T 7/00 (2017.01); H04N 5/225 (2006.01); H04N 5/235 (2006.01); H04N 23/56 (2023.01); H04N 23/74 (2023.01); H04N 5/262 (2006.01)
CPC G06T 5/80 (2024.01) [G02B 21/06 (2013.01); G02B 21/367 (2013.01); G06T 5/50 (2013.01); G06T 7/0002 (2013.01); H04N 23/56 (2023.01); H04N 23/74 (2023.01); G06T 2207/10056 (2013.01); G06T 2207/20081 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/20221 (2013.01); G06T 2207/30168 (2013.01); H04N 5/2628 (2013.01)] 14 Claims
OG exemplary drawing
 
1. A microscope system (100) configured to record images in at least a first and a second imaging mode (501, 502), wherein the microscope system (100) comprises the components of:
a sample imaging objective (1) arranged to provide illumination to a sample space(S) of the microscope system (100) and to collect light (201) from a sample (11) arranged in the sample space(S), wherein an optical axis of the system (OA) aligns with an optical axis (OA0) of the sample imaging objective (1),
an illumination module configured and arranged to emit light (200) to the sample imaging objective (1) for illuminating the sample space(S),
a first reimaging objective (5) and a second reimaging objective (6), wherein an optical axis (OA1) of the first reimaging objective (5) aligns with the optical axis of the system (OA), and wherein the first reimaging objective (5) is arranged and configured to generate an intermediate image of the sample space(S) in an intermediate image space (IS) and wherein the second reimaging objective (6) is arranged and configured to image the intermediate image,
a detection module (70) arranged and configured to detect and record light collected by the second reimaging objective (6), wherein the detection module (70) is arranged along an optical axis (OA2) of the second reimaging objective (6),
an evaluation module (200) comprising a machine learning method (DL), wherein the machine learning method is trained with a first set of images of a sample and a second set of images of the same sample, wherein the first set of images has been acquired in the first imaging mode (501) of the microscope system (100) and wherein the second set has been acquired in the second imaging mode (502) of the microscope system (100), wherein upon acquisition of an image (400) in the second imaging mode (502) the trained machine learning method (DL) is configured to generate and to output a restored image (401) from the image (400) acquired in the second imaging mode, wherein the restored image (401) comprises fewer aberrations than the image (400) acquired in the second imaging mode (52, 53, 57),
wherein the microscope system (100) is configured to adopt a first and a second illumination mode, wherein in the first illumination mode an optical axis of the illumination module and the optical axis (OA0) of the sample imaging objective (1) align and in the second illumination mode an oblique light sheet illumination at the sample space is generated such that illumination propagates along an illumination angle (101) relative to the optical axis (OA0) of the sample imaging objective (1), wherein the microscope system (100) is further configured and arranged to adopt a first detection mode (51) and a second detection mode (52), wherein in the first detection mode (51) the optical axis (OA2) of the second reimaging objective (6) aligns with the optical axis (OA1) of the first reimaging objective (5), wherein in the second detection mode (52) the optical axis (OA2) of the second reimaging objective (6) encloses a reimaging angle (102) with the optical axis (OA1) of the first reimaging objective (5), wherein the first imaging mode (501) comprises a combination of the first or the second illumination mode with the first or the second detection mode (51, 52) and wherein the second imaging mode (502) comprises another combination of the first or the second illumination mode with the first or the second detection mode (51, 52), wherein
the microscope system (100) is configured to adopt a third imaging mode, wherein the third imaging mode (503) comprises yet another combination of the first or the second illumination mode with the first or the second detection mode (51, 52), and wherein upon acquisition of an image (403) in the third imaging mode, the machine learning method (DL) that has been trained with the first set and the second set of images acquired in the first or second imaging mode (501, 502) is configured to generate and to output a restored image (401) from the image (403) acquired in the third imaging mode, wherein the restored image (401) comprises fewer aberrations and/or a higher optical resolution than the image (403) acquired in the third imaging mode.