US 12,321,603 B2
High bandwidth non-volatile memory for AI inference system
Hsiang-Lan Lung, Ardsley, NY (US); and I-Ting Kuo, Taoyuan (TW)
Assigned to Macronix International Co., Ltd., Hsinchu (TW)
Filed by MACRONIX INTERNATIONAL CO., LTD., Hsinchu (TW)
Filed on Feb. 22, 2023, as Appl. No. 18/112,784.
Prior Publication US 2024/0281142 A1, Aug. 22, 2024
Int. Cl. G06F 3/06 (2006.01); G06F 13/16 (2006.01)
CPC G06F 3/0613 (2013.01) [G06F 3/0659 (2013.01); G06F 3/0679 (2013.01); G06F 13/161 (2013.01); G06F 2213/0062 (2013.01)] 20 Claims
OG exemplary drawing
 
1. An artificial intelligence (AI) inference memory device, comprising:
a logic layer die including channel logic implementing connections between a plurality of channels for conducting data to and from an accelerator core via at least one bus; and
a plurality of non-volatile memory (NVM) dies storing arrays of weights and stacked vertically one above another, forming a layered vertical stack of NVM dies, each of the NVM dies including at least one memory chip and a plurality of direct vertical connections to a corresponding channel logic in the logic layer, the stacked NVM dies being organized into banks of NVM dies,
wherein the logic layer die includes, for each bank of NVM dies, a respective controller to execute interface commands with one or more NVM dies of the corresponding bank.