US 11,947,935 B2
Custom models for source code generation via prefix-tuning
Colin Bruce Clement, Seattle, WA (US); Neelakantan Sundaresan, Bellevue, WA (US); Alexey Svyatkovskiy, Bellevue, WA (US); Michele Tufano, Bellevue, WA (US); and Andrei Zlotchevski, Brossard (CA)
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC., Redmond, WA (US)
Filed by MICROSOFT TECHNOLOGY LICENSING, LLC., Redmond, WA (US)
Filed on Nov. 24, 2021, as Appl. No. 17/535,391.
Prior Publication US 2023/0161567 A1, May 25, 2023
Int. Cl. G06F 9/44 (2018.01); G06F 8/35 (2018.01); G06N 3/084 (2023.01); G06N 3/048 (2023.01)
CPC G06F 8/35 (2013.01) [G06N 3/084 (2013.01); G06N 3/048 (2023.01)] 20 Claims
OG exemplary drawing
 
1. A system comprising:
a processor; and a memory that stores a program configured to be executed by the processor, the program including instructions that when executed perform acts that:
access a pre-trained deep learning model trained to generate source code, wherein the pre-trained deep learning model includes a plurality of model parameters, wherein the pre-trained deep learning model comprises an input layer, one or more transformer blocks and an output layer;
tune the pre-trained deep learning model to generate a first custom model through application of a first tuning dataset to the pre-trained deep learning model, wherein the first tuning dataset includes a first prefix that includes a plurality of trainable parameters distinct from the plurality of model parameters, wherein the tuning of the pre-trained deep learning model optimizes the plurality of trainable parameters with the plurality of model parameters frozen, wherein the input layer and the output layer are tuned in a first execution environment of a user space and the one or more transformer blocks are tuned in a second execution environment of a model space, wherein the first execution environment and the second execution environment differ; and
output the first custom model for deployment in an inference system to generate the source code.