US 11,958,198 B2
Predictive instruction text with virtual lab representation highlighting
Jeff Washington, Round Rock, TX (US); Geoffrey J. Budd, Redwood City, CA (US); Nikhita Singh, Palo Alto, CA (US); Jake Sganga, Los Angeles, CA (US); and Alexander Li Honda, Sunnyvale, CA (US)
Assigned to Artificial, Inc., Palo Alto, CA (US)
Filed by Artificial, Inc., Palo Alto, CA (US)
Filed on Aug. 2, 2021, as Appl. No. 17/392,119.
Claims priority of provisional application 63/060,974, filed on Aug. 4, 2020.
Prior Publication US 2022/0043561 A1, Feb. 10, 2022
Int. Cl. G01N 35/00 (2006.01); B25J 9/16 (2006.01); B25J 13/08 (2006.01); B25J 19/02 (2006.01); G06F 3/04812 (2022.01); G06F 3/04815 (2022.01); G06F 3/04842 (2022.01); G06F 40/10 (2020.01); G06F 40/40 (2020.01); G06N 20/00 (2019.01); G06T 7/50 (2017.01); G06T 7/70 (2017.01); G06T 7/80 (2017.01)
CPC B25J 9/1671 (2013.01) [B25J 9/1605 (2013.01); B25J 9/163 (2013.01); B25J 9/1653 (2013.01); B25J 9/1658 (2013.01); B25J 9/1661 (2013.01); B25J 9/1679 (2013.01); B25J 9/1692 (2013.01); B25J 9/1697 (2013.01); B25J 13/089 (2013.01); B25J 19/023 (2013.01); G01N 35/0099 (2013.01); G06F 3/04812 (2013.01); G06F 3/04815 (2013.01); G06F 3/04842 (2013.01); G06F 40/10 (2020.01); G06F 40/40 (2020.01); G06N 20/00 (2019.01); G06T 7/50 (2017.01); G06T 7/70 (2017.01); G06T 7/80 (2017.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
receiving, by a lab automation system, an instruction from a user to perform a protocol within a lab via an interface including a graphical representation of the lab, the lab including a robot and set of lab equipment 3-D rendered within the graphical representation of the lab;
identifying, by the lab automation system, that a portion of text of the received instruction is an ambiguous term by determining that the portion of the text is mapped to a piece of physical equipment that is not included within the lab;
responsive to identifying the ambiguous term, cross-referencing a database of physical equipment within the lab with the portion of text to identify one or more variant pieces of equipment that correspond to the ambiguous term, wherein the variant pieces of equipment perform a same or similar functionality to the piece of physical equipment;
modifying, by the lab automation system, the interface to include a predictive text interface element listing the one or more variant pieces of equipment corresponding to the ambiguous term;
upon a mouseover of a listed variant piece of equipment within the predictive text interface element, modifying, by the lab automation system, the graphical representation of the lab to highlight the 3-D rendered variant piece of equipment corresponding to the mouseover at a location within the graphical representation of the lab corresponding to a real-world location of the listed variant piece of equipment within the lab; and
upon a selection of the listed variant piece of equipment within the predictive text interface element, modifying, by the lab automation system, the instruction to include the listed variant piece of equipment.