	US 12,277,487 B2
	Energy-based associative memory neural networks
Sergey Bartunov, London (GB); Jack William Rae, London (GB); Timothy Paul Lillicrap, London (GB); and Simon Osindero, London (GB)
Assigned to DeepMind Technologies Limited, London (GB)
Appl. No. 17/441,463
Filed by DeepMind Technologies Limited, London (GB)
PCT Filed May 19, 2020, PCT No. PCT/EP2020/063971 § 371(c)(1), (2) Date Sep. 21, 2021, PCT Pub. No. WO2020/234299, PCT Pub. Date Nov. 26, 2020.
Claims priority of provisional application 62/852,196, filed on May 23, 2019.
Prior Publication US 2022/0180147 A1, Jun. 9, 2022
Int. Cl. G06N 3/04 (2023.01)

CPC G06N 3/04 (2013.01)

20 Claims

1. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to implement:

an associative memory neural network having a plurality of associative memory parameters that is configured to process an input to generate an output that defines an energy corresponding to the input, wherein the energy corresponding to the input characterizes a likelihood that the input is stored in the associative memory parameters of the associative memory neural network, wherein the plurality of associative memory parameters are stored in one or more physical computer storage media; and

a reading subsystem that is configured to perform operations comprising:

receiving a given input;

retrieving a target input that is associated with the given input from the associative memory neural network by adjusting the given input using the associative memory neural network, comprising, at each of one or more iterations:

processing the given input, using the associative memory neural network and in accordance with current values of the associative memory parameters, to generate an energy corresponding to the given input;

determining a gradient of a reading objective function that depends on the energy corresponding to the given input with respect to the given input; and

using the gradient to adjust the given input; and

providing the given input after a final iteration of the one or more iterations as the target input that is associated with the given input; and

a writing subsystem that is configured to perform operations comprising:

receiving one or more inputs to be stored;

storing the inputs using the associative memory neural network, comprising, at each of one or more iterations:

processing each of the inputs, using the associative memory neural network and in accordance with current values of the plurality of associative memory parameters, to generate a respective energy corresponding to each of the inputs;

determining gradients of a writing objective function that depends on the energy corresponding to each of the inputs with respect to a specified subset of the associative memory parameters; and

using the gradients to adjust the current values of the specified subset of the associative memory parameters;

wherein the writing objective function depends on a respective gradient of the energy corresponding to each of the inputs with respect to the specified subset of the associative memory parameters; and

wherein the writing objective function depends on a respective magnitude of the gradient of the energy corresponding to each of the inputs with respect to the specified subset of associative memory parameters.