US 11,699,297 B2
Image analysis based document processing for inference of key-value pairs in non-fixed digital documents
Mingfei Gao, Sunnyvale, CA (US); Zeyuan Chen, Mountain View, CA (US); Le Xue, Mountain View, CA (US); Ran Xu, Mountain View, CA (US); and Caiming Xiong, Menlo Park, CA (US)
Assigned to Salesforce, Inc., San Francisco, CA (US)
Filed by salesforce.com, inc., San Francisco, CA (US)
Filed on Jan. 4, 2021, as Appl. No. 17/140,987.
Prior Publication US 2022/0215195 A1, Jul. 7, 2022
Int. Cl. G06V 30/413 (2022.01); G06F 40/186 (2020.01); G06F 40/289 (2020.01); G06V 30/412 (2022.01); G06F 40/295 (2020.01); G06V 30/10 (2022.01); G06V 10/40 (2022.01)
CPC G06V 30/413 (2022.01) [G06F 40/186 (2020.01); G06F 40/289 (2020.01); G06V 30/412 (2022.01); G06F 40/295 (2020.01); G06V 10/40 (2022.01); G06V 30/10 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A computer-implemented method, comprising:
receiving, from a client device, a form image;
obtaining a template indicating one or more fields to extract from the form image,
wherein at least one field is associated with a set of candidate keys for the field;
obtaining, from the form image, a set of phrases and obtaining locations of phrases; and
for the at least one field:
determining key scores for phrases from the set of phrases, a key score for a phrase indicating a likelihood that the phrase is a key for the field on the form,
identifying a set of candidate values for the field from the set of phrases,
determining a set of candidate value-neighbor pairs by identifying a set of neighbors for candidate values from the set of phrases,
determining neighbor scores for the set of candidate value-neighbor pairs, each neighbor score for each candidate value-neighbor pair determined from a combination of the key score for the neighbor of the candidate value-neighbor pair and a spatial relationship of the neighbor to the candidate value for the candidate value-neighbor pair,
selecting a candidate value-neighbor pair associated with a neighbor score above a threshold, and
setting the selected candidate value as a value of the field and the selected neighbor as the key of the field.