US 12,333,792 B2
Guided workflow for deep learning error analysis
Mark William Sabini, River Edge, NJ (US); Kai Yang, Fremont, CA (US); Andrew Yan-Tak Ng, Camas, WA (US); Daniel Bibireata, Bellevue, WA (US); Dillon Laird, Santa Monica, CA (US); Whitney Blodgett, San Francisco, CA (US); Yan Liu, Dalian (CN); Yazhou Cao, San Carlos, CA (US); Yuxiang Zhang, Shanghai (CN); Gregory Diamos, Menlo Park, CA (US); YuQing Zhou, Palo Alto, CA (US); Sanjay Boddhu, Aurora, IL (US); Quinn Killough, Sonoma, CA (US); Shankaranand Jagadeesan, San Jose, CA (US); Camilo Zapata, Bogota (CO); and Sebastian Rodriguez, Medellin (CO)
Assigned to LandingAI Inc., Palo Alto, CA (US)
Filed by LandingAI Inc., Palo Alto, CA (US)
Filed on Oct. 21, 2022, as Appl. No. 17/971,063.
Claims priority of provisional application 63/273,830, filed on Oct. 29, 2021.
Prior Publication US 2023/0136672 A1, May 4, 2023
Int. Cl. G06V 10/776 (2022.01); G06V 10/25 (2022.01); G06V 10/778 (2022.01)
CPC G06V 10/776 (2022.01) [G06V 10/25 (2022.01); G06V 10/7788 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
identifying an incorrectly classified image outputted from a machine learning model;
identifying, using a Neural Template Matching (NTM) model, an additional image that is correlated to the incorrectly classified image, wherein the NTM model outputs correlated images based on a given image and a selection by a user through a user interface of a region of interest (ROI) of the given image, the region defined by a bounding polygon input by the user, wherein the correlated images include features correlated to features within the ROI in the given image;
prompting, through the user interface, a task associated with the additional image;
receiving a response for the task from the user, through the user interface, the response including an indication that the additional image is incorrectly labeled and including a replacement label;
prompting, through the user interface, a second task to select an error type for further investigation from a presented group of error types, wherein the group of error types to present is selected based on a ranking of loss associated with each error type, the error type associated with a category of incorrectly classified images; and
instructing that the machine learning model be retrained using an updated training dataset that includes the replacement label.