US 12,141,553 B2
Programmatically generating evaluation data sets for code generation models
Praphruetpong Athiwaratkun, Jersey City, NJ (US); Zixuan Lin, Seattle, WA (US); Ramana Keerthi, Sammamish, WA (US); Zijian Wang, San Jose, CA (US); Yuchen Tian, Santa Clara, CA (US); Hantian Ding, New York, NY (US); Sri Ranga Akhilesh Bontala, Bothell, WA (US); Matthew Lee, Elmhurst, NY (US); Yanitsa Donchev, Kirkland, CA (US); Ramesh M Nallapati, Fremont, CA (US); Parminder Bhatia, Kearny, NJ (US); Andrew Oliver Arnold, New York, NY (US); Bing Xiang, Mount Kisco, NY (US); Sudipta Sengupta, Sammamish, WA (US); Rama Krishna Sandeep Pokkunuri, Redmond, WA (US); Srinivas Iragavarapu, Redmond, WA (US); Atul Deo, Kirkland, WA (US); and Ankur Deepak Desai, Redmond, WA (US)
Assigned to Amazon Technologies, Inc., Seattle, WA (US)
Filed by Amazon Technologies, Inc., Seattle, WA (US)
Filed on Jun. 22, 2022, as Appl. No. 17/847,113.
Prior Publication US 2023/0418566 A1, Dec. 28, 2023
Int. Cl. G06F 8/33 (2018.01); G06F 8/41 (2018.01); G06F 11/36 (2006.01)
CPC G06F 8/33 (2013.01) [G06F 8/447 (2013.01); G06F 11/3608 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising:
at least one processor; and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement a programming language conversion system, configured to:
receive, via an interface of the programming language conversion system, a request to convert an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests for a code generation system;
convert individual ones of the different items of the data set into a second programming language:
convert a function signature of the item in the first programming language to the second programming language;
convert one or more test statements of the item in the first programming language to the second programming language;
send a request to the code generation system to generate a body of the converted function signature in the second programming language according to a prompt in the item; and
receive the body of the function signature in the second programming language from the code generation system; and
store the converted individual ones of the different items of the evaluation data set as part of a new evaluation data set.