US 12,436,745 B2
Developing a programming language model for machine learning tasks
Mahinthan Chandramohan, Brisbane (AU); Behnaz Hassanshahi, Brisbane (AU); Padmanabhan Krishnan, Brisbane (AU); and Dai Nguyen, Kelvin Grove (AU)
Assigned to Oracle International Corporation, Redwood Shores, CA (US)
Filed by Oracle International Corporation, Redwood Shores, CA (US)
Filed on Oct. 19, 2023, as Appl. No. 18/382,018.
Prior Publication US 2025/0130780 A1, Apr. 24, 2025
Int. Cl. G06F 8/35 (2018.01); G06F 40/284 (2020.01); G06F 40/30 (2020.01)
CPC G06F 8/35 (2013.01) [G06F 40/284 (2020.01); G06F 40/30 (2020.01)] 20 Claims
OG exemplary drawing
 
1. A method comprising:
adjusting a token list to include a language token used by a tokenizer for a pretrained language model,
wherein the pretrained language model comprises a set of layers,
wherein the set of layers comprises a set of initial layers, an embedding layer, and an output layer, and
wherein the output layer generates an output vector from an embedding vector generated by the embedding layer;
performing an output layer modification of the output layer to replace the output vector with the embedding vector;
freezing the set of initial layers to generate a set of frozen layers of the pretrained language model that do not update during training; and
training the pretrained language model using the language token, the output layer modification, and the set of frozen layers to form a fine-tuned model from the pretrained language model, wherein training the pretrained language model comprises backpropagating a difference between a training output vector and an expected vector to a set of end layers of the pretrained language model to form the fine-tuned model.