In the following, to impose a pattern on a network means to make the network memorize the pattern. To recall means to propagate (as in Hopfield Network Simulator).
Suppose we have a Hopfield Network with only four nodes, arranged in a 2 x 2 array. Suppose we impose the pattern that has the top two nodes on and the bottom two off.
Draw a diagram showing all six connections between the nodes and the strength of each connection after the above pattern is imposed.
Suppose we now create the pattern with the top left node on, the other three off, and click Recall. Prove that the network will recall the above pattern after one iteration, and further iterations will not change the pattern.
Suppose we now create the pattern with the top left node off, the other three on, and click Recall. Prove that the network will recall the complement of the above pattern after one iteration, and further iterations will not change the pattern. ("Complement" means replace on with off and vice versa.)
Suppose we now impose the pattern with the two left-hand side nodes on, and the two right-hand side nodes off (without clearing the first pattern imposed above). Give the strength of each connection.
Suppose we try to simulate the above 2x2 network on the 3x3 network at Hopfield Network Simulator by using only the four nodes in the top left of the 3x3 grid or only its four corners. Explain why using just four nodes of the 3x3 array will not properly simulate a 2x2 array.
Suppose we now create on the 2x2 network the pattern with the top left node on, the other three off, and click Recall. What will happen after clicking Recall once? After clicking it twice? Thrice? Can you explain what's going on, and why?
Suppose we have a Hopfield Network with n^2 nodes, arranged in an n x n array. Let P1 and P2 be two different n x n patterns. Let C1 denote the set of all connection strengths that we'd get if we imposed P1 alone on our network (without imposing P2.) Let C2 denote the set of all connection strengths that we'd get if we imposed P2 alone on our network (without imposing P1). Finally, let C denote the set of connection strengths we’d get if we imposed both P1 and P2 on our network.
Is C the sum of C1 and C2? More precisely, is each connection strength in C the sum of the two corresponding strengths in C1 and C2? (If you find this too abstract, as a warm-up try it with a concrete example, say with the 2 x 2 network in the problem above).
Let Q1 be an n x n pattern that's very similar to P1 but not to P2. If we create Q1 on our network (with P1 and P2 both already imposed on it) and click Recall enough times, the network will probably recall P1. Can you explain why this happens even though C might be very different from C1? (In other words, if only P1 is imposed on the network, it's not too surprising that the network recalls P1 when it's shown a pattern Q1 which is similar to P1; but why does this still work even after we "mess up" C1 and change it to C?) Hint: Let V(d, Q1, C) denote the total input that a given node d would get from all the other nodes in Q1 with connection strengths currently being C. Step 1: Show that V(d, Q1, C) = V(d, Q1, C1) + V(d, Q1, C2). Step 2: Argue (informally) that V(d, Q1, C2) is probably small in magnitude compared to V(d, Q1, C1).
Suppose we impose P2 on our network many times, but impose P1 only once. Now if we create Q1 on our network, do we still expect the network to recall P1 rather than P2? Before you try this out on the Hopfield Network simulator, try to figure this out by just thinking! Then explain why it's the way it is.
A Hopfield Network cannot memorize too many patterns; can you explain why? In particular, what's wrong with the following line of reasoning: Suppose we impose a large number of patterns, P1, P2, ... , Pk, on an n x n Hopfield Network (say k is much larger than n^2). Let Q1 be an n x n pattern that's very similar to P1 but not to the other patterns. Let Ci be the connection strengths corresponding to Pi. As in part (b), for any given node d, we can expect V(d, Q1, Ci) to be small for every i other than 1. Furthermore, probably for about half the i's, V(d, Q1, Ci) is negative and for the other i's it's positive. So we can expect the sum V(d, Q1, C2) + ... + V(d, Q1, Ck) to have small magnitude compared to V(d, Q1, C1). So when we create Q1 on the network and click Recall, P1 should be recalled with high probability no matter how large k is. Hint: Suppose you add up 100 numbers, each of which is either 1 or -1 with probability 1/2 each. Now suppose you do the same experiment but with 10^6 numbers instead of 100. In which case is the sum more likely to have absolute value larger than, say, 20?
(Optional) Which of the following "phenomena" or "characteristics" do you think occurs in or applies to Hopfield Networks, human memory, both, or neither? Why? (If you think of other interesting characteristics I could list here, I’d love to hear them.)
Partial recall.
Repetition leads to better recall.
"Nice" or symmetrical patterns are easier to memorize and recall.
Similar pictures are more likely to be confused during recall.
Recognition is easier than recall (e.g., multiple choice is easier than coming up with the answer).
Memory is not recorded locally, but is "spread out" all over the network.
Having a sense of which memories are old vs. new (recorded long ago vs. recently).
Complete recall may take several iterations (several attempts); or it may never be achieved.