US 11,755,915 B2
System and method for quality assurance of media analysis
Jeffrey Fenchel, San Francisco, CA (US); Andras Benke, Bellevue, WA (US); Alex Smith, Chadron, NE (US); Michael Kramer, San Francisco, CA (US); Loretta Jimenez, San Francisco, CA (US); Fabien Vives, San Francisco, CA (US); Julian Alcala, Berkeley, CA (US); and Jonathan R Dodson, San Francisco, CA (US)
Assigned to ZIGNAL LABS, INC., San Francisco, CA (US)
Filed by ZIGNAL LABS, INC., San Francisco, CA (US)
Filed on Jun. 12, 2019, as Appl. No. 16/438,751.
Claims priority of provisional application 62/684,210, filed on Jun. 13, 2018.
Prior Publication US 2019/0385062 A1, Dec. 19, 2019
Int. Cl. G06N 3/084 (2023.01); G06F 16/483 (2019.01); G06N 3/04 (2023.01); G06Q 30/0251 (2023.01); G06Q 50/00 (2012.01); G06Q 10/0639 (2023.01); G06F 40/289 (2020.01)
CPC G06N 3/084 (2013.01) [G06F 16/483 (2019.01); G06N 3/04 (2013.01); G06Q 10/0639 (2013.01); G06Q 30/0271 (2013.01); G06Q 50/01 (2013.01); G06F 40/289 (2020.01)] 25 Claims
OG exemplary drawing
 
1. A system for determining quality level of labeling of a plurality of multimedia data items, wherein the labels relate to reputation polarity, wherein reputation polarity comprises determining an impact of a statement on a reputation of a brand, perception of a brand, or a combination thereof; wherein said brand comprises a person, company, organization or other identifiable entity; the system comprising a plurality of worker computational devices for labeling the items, each worker computational device comprising a processor for executing instructions for labeling the items and a memory for storing the instructions; an analysis computational device for analyzing the labels to determine accuracy by calculating cross polarity accuracy scores and neutral bias scores for a plurality of labels for each item, and for providing feedback through said worker computational devices to increase accuracy of labeling; wherein said cross polarity accuracy score is determined at least partially according to a number of correct labels determined at each worker computational device; wherein said neutral bias score is at least partially determined according to a number of correct reputation polarity labels determined at each worker computational device; wherein said analysis computational device comprises an analysis processor for executing instructions for analyzing the labels and for providing feedback, and an analysis memory for storing the instructions; and a computer network for communication between said computational devices;
the system further comprising a model computational device for being trained on labeled multimedia data items to train a model to automatically determine reputation polarity by classifying an unlabeled multimedia data item, wherein said model computational device comprises a model processor for executing a plurality of instructions for automatically training a model to automatically determine reputation polarity of an unlabeled multimedia data item, and a model memory for storing said instructions; wherein said model to automatically determine reputation polarity receives said plurality of labeled multimedia data items and classifies said multimedia data items according to reputation polarity; wherein an error in correctly classifying said multimedia data items by said model is calculated and is used to adjust said model; wherein said training is repeated until a sufficiently low error is reached.