US 12,013,346 B2
System and method for detection of mobile device fault conditions
William Fitzgerald, Cork (IE); Donal O'Shaughnessy, Cork (IE); Donie Kelly, Cork (IE); Paul O'Sullivan, Cork (IE); Liam O'Callaghan, Cork (IE); James Donovan, Cork (IE); Shane Hallinan, Cork (IE); Charlie McGrory, Cork (IE); Wayne Morgan, Cork (IE); Uday Chitturi, Cork (IE); Patrick Conway, Cork (IE); John Mollaghan, Cork (IE); and Kevin Sutton, Cork (IE)
Assigned to BLANCCO TECHNOLOGY GROUP IP OY, Joensuu (FI)
Filed by BLANCCO TECHNOLOGY GROUP IP OY, Joensuu (FI)
Filed on Dec. 27, 2021, as Appl. No. 17/562,719.
Application 17/562,719 is a division of application No. 15/899,339, filed on Feb. 19, 2018, granted, now 11,210,777.
Prior Publication US 2022/0122243 A1, Apr. 21, 2022
Int. Cl. G01N 21/88 (2006.01); G01N 21/95 (2006.01); G01N 21/958 (2006.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01); G06N 3/084 (2023.01); G06T 7/00 (2017.01); G06T 7/13 (2017.01); G06T 7/80 (2017.01); G06V 10/42 (2022.01); H04N 7/18 (2006.01); G06Q 10/30 (2023.01)
CPC G01N 21/8851 (2013.01) [G01N 21/95 (2013.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06N 3/084 (2013.01); G06T 7/0004 (2013.01); G06T 7/001 (2013.01); G06T 7/13 (2017.01); G06T 7/80 (2017.01); G06V 10/42 (2022.01); H04N 7/18 (2013.01); G01N 2021/8854 (2013.01); G01N 2021/8887 (2013.01); G01N 2021/8893 (2013.01); G01N 2021/9513 (2013.01); G01N 21/958 (2013.01); G01N 2201/0221 (2013.01); G06Q 10/30 (2013.01); G06T 7/0002 (2013.01); G06T 2207/10024 (2013.01); G06T 2207/20084 (2013.01); G06T 2207/30121 (2013.01); G06T 2207/30244 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system comprising:
a mobile device, the device comprising:
a processor in communication with a memory of the mobile device;
a user interface in communication with the processor, the user interface including a touch-sensitive display and a data entry interface; and
a communications module in communication with the processor and configured to provide a communications interface to a host server, the host server including a server processor communicatively connected to a database, a server communications interface, a server user interface, and a server memory;
wherein the memory of the mobile device includes instructions that when executed by the processor cause the mobile device to perform the steps of:
prompting a user of the mobile device to place the mobile device so that a display section of the mobile device faces a reflective surface of a mirror;
performing a tracking calibration function to adjust the mobile device position and display settings for optimal image capture;
modifying a brightness setting and a camera sensitivity setting of the mobile device to optimize image capture; and
determining that the mobile device is in an acceptable orientation with respect to the mirror, and thereupon:
capturing an image of the mobile device as reflected from the mirror; and
formatting the captured image for transmission to the server for defect analysis;
wherein the server memory includes instructions that when executed by the server processor cause the server to perform the steps of:
receiving the formatted image from the server communications interface;
extracting, from the received image, a subimage corresponding to a screen of the mobile device;
transforming the perspective of the subimage into a rectangular aspect;
resampling the transformed subimage to a predetermined input image data configuration;
presenting the resampled transformed subimage to a pre-trained neural network to identify one or more device defects; and
obtaining from the neural network an indication of a device defect that corresponds to the screen of the mobile device;
wherein obtaining from the neural network an indication of a defect further comprises obtaining, from the neural network, an indicia, created through class activation mapping functions, showing a portion of the screen of the mobile device that activates a defective class in the pre-trained neural network.