US 11,721,414 B2
Importing structured prescription records from a prescription label on a medication package
Walter R. Smith, Seattle, WA (US); Chi-Kai Chien, Los Altos Hills, CA (US); Christian Hagel-Sorensen, Sammamish, WA (US); Frédéric Médous, San Francisco, CA (US); Peyman Oreizy, Seattle, WA (US); Jonathan Schwartz, San Francisco, CA (US); Brittany Staten, San Francisco, CA (US); and Paul Teixeira, San Francisco, CA (US)
Assigned to WALMART APOLLO, LLC, Bentonville, AR (US)
Filed by WALMART APOLLO, LLC, Bentonville, AR (US)
Filed on Mar. 12, 2015, as Appl. No. 14/656,691.
Prior Publication US 2016/0267357 A1, Sep. 15, 2016
Int. Cl. G16H 10/60 (2018.01); G16H 20/10 (2018.01); G16Z 99/00 (2019.01); G06V 30/142 (2022.01); G16H 30/20 (2018.01); G06V 30/10 (2022.01); H04N 23/60 (2023.01); G06V 30/14 (2022.01)
CPC G16H 10/60 (2018.01) [G06V 30/142 (2022.01); G16H 20/10 (2018.01); G16H 30/20 (2018.01); G16Z 99/00 (2019.02); G06V 30/10 (2022.01); G06V 30/141 (2022.01); H04N 23/64 (2023.01)] 30 Claims
OG exemplary drawing
 
1. A method implemented via execution of computing instructions configured to run on one or more processors and stored at one or more non-transitory computer-readable media, the method comprising:
during image capture by a camera on a mobile device, presenting, by a medication management system, one or more instructions to a user operating the camera on the mobile device to use the camera to take a set of images of a prescription label on a medication package, wherein the set of images comprises portions of textual characters;
after applying one or more filters to each respective image in the set of images, determining a respective edge density associated with the each respective image in the set of images, wherein an application on the mobile device of the user displays information to the user via a user interface on the mobile device describing a value of the respective edge density for the each respective image in the set of images when the value does not exceed a first threshold measurement, and wherein the application on the mobile device of the user displays information to the user via the user interface on the mobile device that the each respective image is unfocused when the value of the respective edge density for the each respective image exceeds the first threshold measurement;
receiving, by the medication management system, the set of images captured by the camera from the application on the mobile device of the user, wherein the set of images comprises the prescription label on the medication package, and wherein the each respective image in the set of images comprises a respective different portion of the prescription label;
determining, using contrast, a location of the each respective image in the set of images that is associated with a respective portion of the prescription label on a non-planar surface of the medical package;
determining, by the medication management system, one or more respective quality measurements for the each respective image in the set of images, wherein each respective quality measurement of the one or more respective quality measurements is associated with one or more respective adjustments in the each respective image;
responsive to a quality measurement in the one or more respective quality measurements not exceeding a second threshold measurement associated with an adjustment of the one or more respective adjustments associated with the quality measurement:
providing information for display on the user interface of the mobile device that describes the quality measurement to the user; and
providing for display on the user interface a request for a respective replacement image to replace the each respective image having the quality measurement, wherein replacement images are requested until the quality measurement exceeds the second threshold measurement, and wherein the replacement images comprise the respective replacement image;
responsive to the one or more respective quality measurements exceeding corresponding threshold measurements associated with the one or more respective adjustments for the one or more respective quality measurements:
determining, for the each respective image in the set of images of the medication package, structural information identifying (i) an object similar in shape to the medication package captured in the each respective image, and (ii) the non-planar surface of the medication package in the each respective image;
upon identifying that the non-planar surface of the medication package exists, determining one or more transform functions for the non-planar surface of the medication package based on the structural information of the medication package captured in the each respective image, the one or more transform functions operable for flattening a different portion of the prescription label on the non-planar surface based on the object identified by the structural information, wherein the one or more transform functions map a first pixel on the different portion of the prescription label on the non-planar surface of the medication package to a second pixel on a first flat surface; and
reconstructing the different portion of the prescription label on the non-planar surface of the medication package as a flattened reconstruction of the different portion of the prescription label using the set of images and the one or more transform functions; and
providing, by the medication management system, for display on the user interface a reconstructed prescription label for creation of a prescription record for the user based on extracted information from the medication package, wherein the reconstructed prescription label is based at least in part on the flattened reconstruction of the different portion of the prescription label.