US 12,443,420 B2
Automatic image conversion
Michael Sollami, Cambridge, MA (US); Sönke Rohde, San Francisco, CA (US); Alan Martin Ross, San Francisco, CA (US); David James Woodward, Bozeman, MT (US); Jessica Lundin, Seattle, WA (US); Owen Winne Schoppe, Orinda, CA (US); Brian J. Lonsdorf, Soquel, CA (US); and Aashish Jain, Cambridge, MA (US)
Assigned to Salesforce, Inc., San Francisco, CA (US)
Filed by salesforce.com, inc., San Francisco, CA (US)
Filed on Jan. 26, 2022, as Appl. No. 17/649,045.
Claims priority of provisional application 63/271,968, filed on Oct. 26, 2021.
Prior Publication US 2023/0129240 A1, Apr. 27, 2023
Int. Cl. G06N 3/08 (2023.01); G06F 3/04845 (2022.01); G06F 8/38 (2018.01); G06F 9/451 (2018.01); G06F 9/54 (2006.01); G06N 3/044 (2023.01); G06N 3/045 (2023.01); G06V 10/762 (2022.01); G06V 10/771 (2022.01); G06V 10/82 (2022.01); G06N 3/047 (2023.01); G06N 3/088 (2023.01); G06N 7/01 (2023.01); G06V 30/19 (2022.01)
CPC G06F 9/451 (2018.02) [G06F 3/04845 (2013.01); G06F 9/547 (2013.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06V 10/763 (2022.01); G06V 10/771 (2022.01); G06V 10/82 (2022.01); G06F 8/38 (2013.01); G06N 3/044 (2023.01); G06N 3/047 (2023.01); G06N 3/088 (2013.01); G06N 7/01 (2023.01); G06V 30/19173 (2022.01)] 20 Claims
OG exemplary drawing
 
1. A method, comprising:
generating, by a generative adversarial network (GAN) executed by a server computer system, a layout image, wherein the layout image is generated by inputting, into the GAN, a rendered existing user interface (UI) design and a separate background image that includes a picture upon which one or more UI elements to be included in a new UI design will be overlayed, wherein the layout image includes a plurality of portions representing a plurality of UI elements included in the new UI design, wherein the new UI design is generated for the same platform as the existing UI design, and wherein respective ones of the plurality of portions included in the layout image are pixels that include multiple properties representing attributes of respective ones of the UI elements;
transforming, by a trained residual neural network (ResNet) executed by the server computer system, the layout image to a text-based representation of the layout image that specifies coordinates of bounding regions of the plurality of UI elements included in the new UI design; and
generating, by the server computer system based on the text-based representation, program code executable to render the new UI design.