Academia.eduAcademia.edu

Figure 7 – uploaded by David Alejandro

See full PDFdownloadDownload figure
FIGURE 6. A paradigm for the solutions to the parity problem discovered by the learning system. See text for ex- planation.

Figure 6 A paradigm for the solutions to the parity problem discovered by the learning system. See text for ex- planation.

Related Figures (28)

FIGURE }. A moultilayer network. In this case the information coming to the input units ie recoded into an ioter- nal representation and the outputs are generated by the internal representation rather than by the original pattern. Input patterns can always be encoded, if there are enough hidden units, io a form so that the appropriate output pat- tern can be generated from any input pattern. VesW TSS SSR SO Ew org SH COW Ee ws wyVeweee Vw Sw BU OW UE fw ewe Swtese Minsky and Papert (1969) have provided a very careful analysis of conditions under which such systems are capable of carrying out the required mappings. They show that in a large number of interesting cases, networks of this kind are incapable of solving the problems. On the other hand, as Minsky and Papert also pointed out, if there is a layer of simple perceptron- like hidden units, as shown in Figure 1, with which the original input pattern can be aug- mented, there is always a recoding (i.e., an internal representation) of the input patterns in the hidden units in which the similarity of the patterns among the hidden units can support any required mapping from the input to the output units. Thus, if we have the right connections from the input units to a large enough set of hidden units, we can always find a representation
FIGURE }. A moultilayer network. In this case the information coming to the input units ie recoded into an ioter- nal representation and the outputs are generated by the internal representation rather than by the original pattern. Input patterns can always be encoded, if there are enough hidden units, io a form so that the appropriate output pat- tern can be generated from any input pattern. VesW TSS SSR SO Ew org SH COW Ee ws wyVeweee Vw Sw BU OW UE fw ewe Swtese Minsky and Papert (1969) have provided a very careful analysis of conditions under which such systems are capable of carrying out the required mappings. They show that in a large number of interesting cases, networks of this kind are incapable of solving the problems. On the other hand, as Minsky and Papert also pointed out, if there is a layer of simple perceptron- like hidden units, as shown in Figure 1, with which the original input pattern can be aug- mented, there is always a recoding (i.e., an internal representation) of the input patterns in the hidden units in which the similarity of the patterns among the hidden units can support any required mapping from the input to the output units. Thus, if we have the right connections from the input units to a large enough set of hidden units, we can always find a representation
their own internal representations of the input patterns. It is interesting to note that had the input patterns contained a third input taking the value 1 whenever the first two have value 1 as shown in Table 2, a two-layer system would be able to solve the problem.
their own internal representations of the input patterns. It is interesting to note that had the input patterns contained a third input taking the value 1 whenever the first two have value 1 as shown in Table 2, a two-layer system would be able to solve the problem.
FIGURE 2. A simple XOR network with one hidden unit. See text for explanation. The existence of networks such as this illustrates the potential power of hidden units and internal representations. The problem, as noted by Minsky and Papert, is that whereas there is a very simple guaranteed learning rule for all problems that can be solved without hidden units, namely, the perceptron convergence procedure (or the variation due originally to Widrow and Hoff, 1960, which we call the delta rule; see Chapter 11), there is no equally powerful rule for learning in networks with hidden units. There have been three basic responses to this lack. One response is represented by competitive learning (Chapter 5) in which simple unsupervised learning rules are employed so that useful hidden units develop. Although these approaches are promising, there is no external force to insure that hidden units appropriate for the required mapping are developed. The second response is to simply assume an internal representation that, on some a priori grounds, seems reasonable. This is the tack taken in the chapter on verb learning (Chapter 18) and in the interactive activation model of word perception (McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982). The third approach is to attempt to develop a learning procedure capable of learning an internal representation adequate for per- forming the task at hand. One such development is presented in the discussion of Boltzmann machines in Chapter 7. As we have seen, this procedure involves the use of stochastic units, requires the network to reach equilibrium in two different phases, and is limited to symmetric networks. Another recent approach, also employing stochastic units, has been developed by Barto (1985) and various of his colleagues (cf. Barto & Anandan, 1985). In this chapter we
FIGURE 2. A simple XOR network with one hidden unit. See text for explanation. The existence of networks such as this illustrates the potential power of hidden units and internal representations. The problem, as noted by Minsky and Papert, is that whereas there is a very simple guaranteed learning rule for all problems that can be solved without hidden units, namely, the perceptron convergence procedure (or the variation due originally to Widrow and Hoff, 1960, which we call the delta rule; see Chapter 11), there is no equally powerful rule for learning in networks with hidden units. There have been three basic responses to this lack. One response is represented by competitive learning (Chapter 5) in which simple unsupervised learning rules are employed so that useful hidden units develop. Although these approaches are promising, there is no external force to insure that hidden units appropriate for the required mapping are developed. The second response is to simply assume an internal representation that, on some a priori grounds, seems reasonable. This is the tack taken in the chapter on verb learning (Chapter 18) and in the interactive activation model of word perception (McClelland & Rumelhart, 1981; Rumelhart & McClelland, 1982). The third approach is to attempt to develop a learning procedure capable of learning an internal representation adequate for per- forming the task at hand. One such development is presented in the discussion of Boltzmann machines in Chapter 7. As we have seen, this procedure involves the use of stochastic units, requires the network to reach equilibrium in two different phases, and is limited to symmetric networks. Another recent approach, also employing stochastic units, has been developed by Barto (1985) and various of his colleagues (cf. Barto & Anandan, 1985). In this chapter we
FIGURE 3. Observed XOR network. The connection weights are written on the arrows and the biases are written in the circles. Note a positive bias means that the unit is on unless turned off. It is useful to begin with the exclusive-or problem since it is the classic problem requiring hidden units and since many other difficult problems involve an XOR as a subproblem. We have run the XOR problem many times and with a couple of exceptions discussed below, the system has always solved the problem. Figure 3 shows one of the solutions to the problem.
FIGURE 3. Observed XOR network. The connection weights are written on the arrows and the biases are written in the circles. Note a positive bias means that the unit is on unless turned off. It is useful to begin with the exclusive-or problem since it is the classic problem requiring hidden units and since many other difficult problems involve an XOR as a subproblem. We have run the XOR problem many times and with a couple of exceptions discussed below, the system has always solved the problem. Figure 3 shows one of the solutions to the problem.
FIGURE 5. A network at a tocal minimum for the exclusive-or problem. The dated lines indicate negative weights. Note that whenever the right-most input unit is on it tums on bork hidden units. The weights connecting the hiddea units to the output are arranged so that when both hidden units are on, the output unit gets a net input of zero. This feads to an output value of 0.5. In the other cases the network provides the correct answer.
FIGURE 5. A network at a tocal minimum for the exclusive-or problem. The dated lines indicate negative weights. Note that whenever the right-most input unit is on it tums on bork hidden units. The weights connecting the hiddea units to the output are arranged so that when both hidden units are on, the output unit gets a net input of zero. This feads to an output value of 0.5. In the other cases the network provides the correct answer.
discovered by the system. The solid lincs in the figure indicate weights of +1 and the dotted lines indicate weights of —1. The numbers in the circles represent the biases of the units. Basi- cally, the hidden units arranged themselves so that they count the number of inputs. In the diagram, the one at the far left comes on if one or more input units are on, the next comes on if two or more are on, etc. All of the hidden units come on if all of the input lines are on. The first m hidden units come on whenever m bits are on in the input pattern. The hidden units then connect with alternately positive and negative weights. In this way the net input from the hidden units is zero for even numbers and +1 for odd numbers. Table 3 shows the actual solution attained for one of our simulations with four input lines and four hidden units. This solution was reached after 2,825 presentations of each of the sixteen patterns with n = 0.5. Note that the solution is roughly a mirror image of that shown in Figure 6 in that the number of hidden units turned on is equal to the number of zero input values rather than the number of ones. Beyond that the principle is that shown above. It should be noted that the internal representation created by the learning rule is to arrange that the number of hidden units that come on is equal to the number of zeros in the input and that the particular hidden units that come on depend only on the number, not on which input units are on. This is exactly the sort of recoding required by parity. It is not the kind of representation readily discovered by unsupervised learning schemes such as competitive learning.
discovered by the system. The solid lincs in the figure indicate weights of +1 and the dotted lines indicate weights of —1. The numbers in the circles represent the biases of the units. Basi- cally, the hidden units arranged themselves so that they count the number of inputs. In the diagram, the one at the far left comes on if one or more input units are on, the next comes on if two or more are on, etc. All of the hidden units come on if all of the input lines are on. The first m hidden units come on whenever m bits are on in the input pattern. The hidden units then connect with alternately positive and negative weights. In this way the net input from the hidden units is zero for even numbers and +1 for odd numbers. Table 3 shows the actual solution attained for one of our simulations with four input lines and four hidden units. This solution was reached after 2,825 presentations of each of the sixteen patterns with n = 0.5. Note that the solution is roughly a mirror image of that shown in Figure 6 in that the number of hidden units turned on is equal to the number of zero input values rather than the number of ones. Beyond that the principle is that shown above. It should be noted that the internal representation created by the learning rule is to arrange that the number of hidden units that come on is equal to the number of zeros in the input and that the particular hidden units that come on depend only on the number, not on which input units are on. This is exactly the sort of recoding required by parity. It is not the kind of representation readily discovered by unsupervised learning schemes such as competitive learning.
Ackley, Hinton, and Sejnowski (1985) have posed a problem in which a set of orthogonal nput patterns are mapped to a set of orthogonal output patterns through a small set of hidden inits. In such cases the internal representations of the patterns on the hidden units must be ather efficient. Suppose that we attempt to map N input patterns onto N output patterns. suppose further that log.N hidden units are provided. In this case, we expect that the system will learn to use the hidden units to form a binary code with a distinct binary pattern for each f the N input patterns. Figure 7 illustrates the basic architecture for the encoder problem. Essentially, the problem is to learn an encoding of an N bit pattern into a log,N bit pattern ind then learn to decode this representation into the output pattern. We have presented the ystem with a number of these problems. Here we present a problem with eight input pat- cerns, eight output patterns, and three hidden units. In this case the required mapping is the dentity mapping illustrated in Table 4. The problem is simply to turn on the same bit in the yutput as in the input. Table 5 shows the mapping generated by our Icarning system on this xample. It is of some interest that the system employed its ability to use intermediate values n solving this problem. It could, of course, have found a solution in which the hidden units
Ackley, Hinton, and Sejnowski (1985) have posed a problem in which a set of orthogonal nput patterns are mapped to a set of orthogonal output patterns through a small set of hidden inits. In such cases the internal representations of the patterns on the hidden units must be ather efficient. Suppose that we attempt to map N input patterns onto N output patterns. suppose further that log.N hidden units are provided. In this case, we expect that the system will learn to use the hidden units to form a binary code with a distinct binary pattern for each f the N input patterns. Figure 7 illustrates the basic architecture for the encoder problem. Essentially, the problem is to learn an encoding of an N bit pattern into a log,N bit pattern ind then learn to decode this representation into the output pattern. We have presented the ystem with a number of these problems. Here we present a problem with eight input pat- cerns, eight output patterns, and three hidden units. In this case the required mapping is the dentity mapping illustrated in Table 4. The problem is simply to turn on the same bit in the yutput as in the input. Table 5 shows the mapping generated by our Icarning system on this xample. It is of some interest that the system employed its ability to use intermediate values n solving this problem. It could, of course, have found a solution in which the hidden units
7IGURE 7. A network for solving the encoder problem. [In this problem there are N orthogonal input patterns cac! aired with one of N orthogonal output patterns. There are onfy fogN, hidden units. Thus, if the hidden units tak mn binary values, the hidden units must form a binary number to encode each of the input patterns. This is exactl vhat the system learns to do.
7IGURE 7. A network for solving the encoder problem. [In this problem there are N orthogonal input patterns cac! aired with one of N orthogonal output patterns. There are onfy fogN, hidden units. Thus, if the hidden units tak mn binary values, the hidden units must form a binary number to encode each of the input patterns. This is exactl vhat the system learns to do.
took on only the values of zero and one. Often it does just that, but in this instance, and many others, there are solutions that use the intermediate values, and the learning system finds them even though it has a bias toward extreme values. It is possible to set up problems that require the system to make use of intermediate values in order to solve a problem. We now turn to such a case. ON ee ee ee ee, a a, oe: oe a Se ee ee eee a See | | ea
took on only the values of zero and one. Often it does just that, but in this instance, and many others, there are solutions that use the intermediate values, and the learning system finds them even though it has a bias toward extreme values. It is possible to set up problems that require the system to make use of intermediate values in order to solve a problem. We now turn to such a case. ON ee ee ee ee, a a, oe: oe a Se ee ee eee a See | | ea
FIGURE 8. The network illustrating the use of intermediate values in solving a problem. See text for explanatioa Our experience is that there is a propensity for the hidden units to take on extreme values, but, whenever the learning problem calls for it, they can learn to take on graded values. It is likely that the propensity to take on extreme values follows from the fact that the logistic is a sigmoid so that increasing magnitudes of its inputs push it toward zero or one. This means that in a problem in which intermediate values are required, the incoming weights must remain of moderate size. It is interesting that the derivation of the generalized delta rule does not depend on all of the units having identical activation functions. Thus, it would be possible for some units, those required to encode information in a graded fashion, to be linear while others might be logistic. The linear unit would have a much wider dynamic range and could encode more different values. This would be a useful role for a linear unit in a network with hidden units.
FIGURE 8. The network illustrating the use of intermediate values in solving a problem. See text for explanatioa Our experience is that there is a propensity for the hidden units to take on extreme values, but, whenever the learning problem calls for it, they can learn to take on graded values. It is likely that the propensity to take on extreme values follows from the fact that the logistic is a sigmoid so that increasing magnitudes of its inputs push it toward zero or one. This means that in a problem in which intermediate values are required, the incoming weights must remain of moderate size. It is interesting that the derivation of the generalized delta rule does not depend on all of the units having identical activation functions. Thus, it would be possible for some units, those required to encode information in a graded fashion, to be linear while others might be logistic. The linear unit would have a much wider dynamic range and could encode more different values. This would be a useful role for a linear unit in a network with hidden units.
FIGURE 9. Network for solving the symmetry problem. The six open circles represent the input units. There are two hidden units, one shown above and one below the input units. The output unit is shown to the far left. See text for explanation. Another interesting problem we studied is that of classifying input strings as to whether or not they are symmetric about their center. We used patterns of various lengths with various numbers of hidden units. To our surprise, we discovered that the problem can always be solved with only two hidden units. To understand the derived representation, consider one of the solutions generated by our system for strings of Iength six. This solution was arrived at after 1,208 presentations of each six-bit pattern with n = 0.1. The final network is shown in Figure 9. For simplicity we have shown the six input units in the center of the diagram with one hidden unit above and one below. The output unit, which signals whether or not the string is symmetric about its center, is shown at the far right. The key point to see about this solution is that for a given hidden unit, weights that are symmetric about the middle are equal in magnitude and opposite in sign. That means that if a symmetric pattern is on, both hidden units will receive a net input of zero from the input units, and, since the hidden units have a negative bias, both will be off. In this case, the output unit, having a positive bias, will be on. The next most important thing to note about the solution is that the weights on each side of the midpoint of the string are in the ratio of 1:2:4. This insures that each of the eight patterns that can occur on each side of the midpoint sends a unique activation sum to the hidden unit. This assures that there is no pattern on the left that will exactly balance a non-mirror-image pattern on the right. Finally, the two hidden units have identical patterns of weights from the input units except for sign. This insures that for every nonsymmetric pattern, at least one of the two hidden units will come on and tum on the output unit. To summarize, the network is arranged so that both hidden units will receive exactly zero activation from the input units when the pattern is symmetric, and at least one of them will receive positive input for every nonsymmetric pattern.
FIGURE 9. Network for solving the symmetry problem. The six open circles represent the input units. There are two hidden units, one shown above and one below the input units. The output unit is shown to the far left. See text for explanation. Another interesting problem we studied is that of classifying input strings as to whether or not they are symmetric about their center. We used patterns of various lengths with various numbers of hidden units. To our surprise, we discovered that the problem can always be solved with only two hidden units. To understand the derived representation, consider one of the solutions generated by our system for strings of Iength six. This solution was arrived at after 1,208 presentations of each six-bit pattern with n = 0.1. The final network is shown in Figure 9. For simplicity we have shown the six input units in the center of the diagram with one hidden unit above and one below. The output unit, which signals whether or not the string is symmetric about its center, is shown at the far right. The key point to see about this solution is that for a given hidden unit, weights that are symmetric about the middle are equal in magnitude and opposite in sign. That means that if a symmetric pattern is on, both hidden units will receive a net input of zero from the input units, and, since the hidden units have a negative bias, both will be off. In this case, the output unit, having a positive bias, will be on. The next most important thing to note about the solution is that the weights on each side of the midpoint of the string are in the ratio of 1:2:4. This insures that each of the eight patterns that can occur on each side of the midpoint sends a unique activation sum to the hidden unit. This assures that there is no pattern on the left that will exactly balance a non-mirror-image pattern on the right. Finally, the two hidden units have identical patterns of weights from the input units except for sign. This insures that for every nonsymmetric pattern, at least one of the two hidden units will come on and tum on the output unit. To summarize, the network is arranged so that both hidden units will receive exactly zero activation from the input units when the pattern is symmetric, and at least one of them will receive positive input for every nonsymmetric pattern.
FIGURE 10. Minimal network for adding two two-bit binary numbers. There are four input units, three output units, and two hidden units. The output patterns can be viewed as the binary representation of the sum of two two- bit binary oumbers represented by the input patterns. The second and fourth input units in the diagram correspond to the low-order bits of the two binary oumbers, and the first and third units correspond to the two higher order bits. The hidden units correspond to the carry bits ia the summation. The hidden unit on the far right comes on when both of the lower order bits in the input pattern are turned on, and the one on the left comes on when both bigher order bits are turned on or when one of the higher order bits and the other hidden unit is turned on. The weights on al! lines are assumed to be +1 except where noted. Negative connections are indicated by dashed lines. As usual, the biases are indicated by the oumbers in the circles. Another interesting problem on which we have tested our learning algorithm is the simple binary addition problem. This problem is interesting because there is a very elegant solution to it, because it is the one problem we have found where we can reliably find local minima and because the way of avoiding these local minima gives us some insight into the conditions under which local minima may be found and avoided. Figure 10 illustrates the basic problem and a minimal solution to it. There are four input units, three output units, and two hidden units. The output pattems can be viewed as the binary representation of the sum of two two-bit binary numbers represented by the input patterns. The second and fourth input units in the diagram correspond to the low-order bits of the two binary numbers and the first and third units correspond to the two hicher order bits. The hidden units corresnond to the carry hire
FIGURE 10. Minimal network for adding two two-bit binary numbers. There are four input units, three output units, and two hidden units. The output patterns can be viewed as the binary representation of the sum of two two- bit binary oumbers represented by the input patterns. The second and fourth input units in the diagram correspond to the low-order bits of the two binary oumbers, and the first and third units correspond to the two higher order bits. The hidden units correspond to the carry bits ia the summation. The hidden unit on the far right comes on when both of the lower order bits in the input pattern are turned on, and the one on the left comes on when both bigher order bits are turned on or when one of the higher order bits and the other hidden unit is turned on. The weights on al! lines are assumed to be +1 except where noted. Negative connections are indicated by dashed lines. As usual, the biases are indicated by the oumbers in the circles. Another interesting problem on which we have tested our learning algorithm is the simple binary addition problem. This problem is interesting because there is a very elegant solution to it, because it is the one problem we have found where we can reliably find local minima and because the way of avoiding these local minima gives us some insight into the conditions under which local minima may be found and avoided. Figure 10 illustrates the basic problem and a minimal solution to it. There are four input units, three output units, and two hidden units. The output pattems can be viewed as the binary representation of the sum of two two-bit binary numbers represented by the input patterns. The second and fourth input units in the diagram correspond to the low-order bits of the two binary numbers and the first and third units correspond to the two hicher order bits. The hidden units corresnond to the carry hire
FIGURE 11. Network found for the summation problem. A: The connections from the input units to the three hidden units and the connections among the hidden units. B: The connections from the input and hidden units to the lowest order output unit. C: The connections from the ioput and hidden units to the middle output unit. D: The connections from the input and hidden units to the highest order output unit. numbers we have found that the system always solves the problem with one extra hidden unit. With larger numbers it may require two or three more. For purposes of illustration, we show the results of one of our runs with three rather than the minimum two hidden units. Figure 11 shows the state reached by the network after 3,020 presentations of each input pattern and with a learning rate of n = 0.5. For convenience, we show the network in four parts. In Fig- ure 11A we show the connections to and among the hidden units. This figure shows the inter- nal representation generated for this problem. The “lowest” hidden unit turns off whenever cither of the low-order bits are on. In other words it detects the case in which no low-order bit is turn on. The “highest” hidden unit is arranged so that it comes on whenever the sum is less than two. The conditions under which the middle hidden unit comes on are more com- plex. Table 8 shows the patterns of hidden units which occur to each of the sixteen input pat- ferns. Figure 11B shows the connections to the lowest order output unit. Noting that the clevant hidden unit comes on when ncither low-order input unit is on, it is clear how the sys- ‘em computes XOR. When both low-order inputs are off, the output unit is turned off by the
FIGURE 11. Network found for the summation problem. A: The connections from the input units to the three hidden units and the connections among the hidden units. B: The connections from the input and hidden units to the lowest order output unit. C: The connections from the ioput and hidden units to the middle output unit. D: The connections from the input and hidden units to the highest order output unit. numbers we have found that the system always solves the problem with one extra hidden unit. With larger numbers it may require two or three more. For purposes of illustration, we show the results of one of our runs with three rather than the minimum two hidden units. Figure 11 shows the state reached by the network after 3,020 presentations of each input pattern and with a learning rate of n = 0.5. For convenience, we show the network in four parts. In Fig- ure 11A we show the connections to and among the hidden units. This figure shows the inter- nal representation generated for this problem. The “lowest” hidden unit turns off whenever cither of the low-order bits are on. In other words it detects the case in which no low-order bit is turn on. The “highest” hidden unit is arranged so that it comes on whenever the sum is less than two. The conditions under which the middle hidden unit comes on are more com- plex. Table 8 shows the patterns of hidden units which occur to each of the sixteen input pat- ferns. Figure 11B shows the connections to the lowest order output unit. Noting that the clevant hidden unit comes on when ncither low-order input unit is on, it is clear how the sys- ‘em computes XOR. When both low-order inputs are off, the output unit is turned off by the
hidden unit. When both low-order input units are on, the output is turned off directly by the two input units. If just one is on, the positive bias on the output unit keeps it on. Figure 11C gives the conncctions to the middle output unit, and in Figure 11D we show those connec- tions to the left-most, highest order output unit. It is somewhat difficult to see how these connections always lead to the correct output answer, but, as can be verified from the figures, the network is balanced so that this works.
hidden unit. When both low-order input units are on, the output is turned off directly by the two input units. If just one is on, the positive bias on the output unit keeps it on. Figure 11C gives the conncctions to the middle output unit, and in Figure 11D we show those connec- tions to the left-most, highest order output unit. It is somewhat difficult to see how these connections always lead to the correct output answer, but, as can be verified from the figures, the network is balanced so that this works.
FIGURE 12. The solution discovered for the negation problem. The right-most unit is the negation unit. The prob- lem has been reduced and solved as three exclusive-ors between the negation unit and each of the other three units. Most of the problems discussed so far (except the symmetry problem) are rather abstract mathematical problems. We now tum to a more geometric problem—that of discriminating between a 7 and a C—independent of translation and rotation. Figure 13 shows the stimulus patterns used in these experiments. Note, these patterns are each made of five squares and differ from one another by a single square. Moreover, as Minsky and Papert (1969) point out, when considering the set of patterns over all possible translations and rotations (of 90°, 180°, and 270°), the patterns do not differ in the set of distances among their pairs of squares. To see a difference between the sets of patterns one must look, at least, at configurations of tri-
FIGURE 12. The solution discovered for the negation problem. The right-most unit is the negation unit. The prob- lem has been reduced and solved as three exclusive-ors between the negation unit and each of the other three units. Most of the problems discussed so far (except the symmetry problem) are rather abstract mathematical problems. We now tum to a more geometric problem—that of discriminating between a 7 and a C—independent of translation and rotation. Figure 13 shows the stimulus patterns used in these experiments. Note, these patterns are each made of five squares and differ from one another by a single square. Moreover, as Minsky and Papert (1969) point out, when considering the set of patterns over all possible translations and rotations (of 90°, 180°, and 270°), the patterns do not differ in the set of distances among their pairs of squares. To see a difference between the sets of patterns one must look, at least, at configurations of tri-
The T-C Problem
The T-C Problem
FIGURE 14. The network for solving ‘he T-C problem. See text for explanation
FIGURE 14. The network for solving ‘he T-C problem. See text for explanation
FIGURE 15. Receptive fields found in different runs of the T-C problem. A: Ao on-center-off-surround receptive field for detecting T's. B: A vertical bar detector which responds to T's more strongly than C's. C: A diagonal bar detector. A T activates five such detectors whereas a C activates only three such detectors. D: A compactness detector. This inhibitory receptive field turns off whenever an input covers any region of its receptive field. Since the C is more compact than the 7 it turns off 20 such detectors whereas the T turns off 21 of them.
FIGURE 15. Receptive fields found in different runs of the T-C problem. A: Ao on-center-off-surround receptive field for detecting T's. B: A vertical bar detector which responds to T's more strongly than C's. C: A diagonal bar detector. A T activates five such detectors whereas a C activates only three such detectors. D: A compactness detector. This inhibitory receptive field turns off whenever an input covers any region of its receptive field. Since the C is more compact than the 7 it turns off 20 such detectors whereas the T turns off 21 of them.
FIGURE 16. The generalized delta rule for sigma-pi units. The products of activation values of individual units ac- tivate output units. See text for explanation of bow the 8 values are computed ia this case.
FIGURE 16. The generalized delta rule for sigma-pi units. The products of activation values of individual units ac- tivate output units. See text for explanation of bow the 8 values are computed ia this case.
FIGURE 17. A comparison of a recurrent network and s feedforward oetwork with identical behavior. A: A com pletely connected recurrent network with two units. 8: A feedforward network which behaves the same as the re current oetwork. In this case, we have a separate unit for each tine step and we require that the weights connection, each layer of units to the oext be the same for all layers. Moreover, they must be the same as the analogous weight io the recurrent case. We have thus far restricted ourselves to feedforward nets. This may seem like a substantial restriction, but as Minsky and Papert point out, there is, for every recurrent network, a feed- forward network with identical behavior (over a finite period of time). We will now indicate how this construction can proceed and thereby show the correct form of the learning rule for the recurrent network. Consider the simple recurrent network shown in Figure 17A. The same network in a feedforward architecture is shown in Figure 17B. The behavior of a
FIGURE 17. A comparison of a recurrent network and s feedforward oetwork with identical behavior. A: A com pletely connected recurrent network with two units. 8: A feedforward network which behaves the same as the re current oetwork. In this case, we have a separate unit for each tine step and we require that the weights connection, each layer of units to the oext be the same for all layers. Moreover, they must be the same as the analogous weight io the recurrent case. We have thus far restricted ourselves to feedforward nets. This may seem like a substantial restriction, but as Minsky and Papert point out, there is, for every recurrent network, a feed- forward network with identical behavior (over a finite period of time). We will now indicate how this construction can proceed and thereby show the correct form of the learning rule for the recurrent network. Consider the simple recurrent network shown in Figure 17A. The same network in a feedforward architecture is shown in Figure 17B. The behavior of a
10 If the constraint that biases be negative is not imposed, other solutions are possible. These solutions can iovolve the units passing through the complements of the shifted pattern or even through more complicated intermediate states. These trajectories are interesting in that they match a simple shift register on all even numbers of shifts, but do not match following an odd number of shifts. Learning to complete sequences. Table 10 shows a set of 25 sequences which were chosen so that the first two items of a sequence uniquely determine the remaining four. We used this set of sequences to test out the learning abilities of a recurrent network. The network consisted of five input units (A, B, C, D, E), 30 hidden units, and three output units (1, 2, 3). At Time 1, the input unit corresponding to the first item of the sequence is turned on and the other input units are turned off. At Time 2, the input unit for the second item in the sequence is turned on and the others are all turned off. Then all the input units are turned off and kept off for the remaining four steps of the forward iteration. The network must learn to make the output units adopt states that represent the rest of the sequence. Unlike simple feedforward networks (or their iterative equivalents), the errors are not only assessed at the final layer or time. The output units must adopt the appropriate states during the forward iteration, and so during the back-propagation phase, errors are injected at each time-step by comparing the remembered actual states of the oufnut units with their desired states.
10 If the constraint that biases be negative is not imposed, other solutions are possible. These solutions can iovolve the units passing through the complements of the shifted pattern or even through more complicated intermediate states. These trajectories are interesting in that they match a simple shift register on all even numbers of shifts, but do not match following an odd number of shifts. Learning to complete sequences. Table 10 shows a set of 25 sequences which were chosen so that the first two items of a sequence uniquely determine the remaining four. We used this set of sequences to test out the learning abilities of a recurrent network. The network consisted of five input units (A, B, C, D, E), 30 hidden units, and three output units (1, 2, 3). At Time 1, the input unit corresponding to the first item of the sequence is turned on and the other input units are turned off. At Time 2, the input unit for the second item in the sequence is turned on and the others are all turned off. Then all the input units are turned off and kept off for the remaining four steps of the forward iteration. The network must learn to make the output units adopt states that represent the rest of the sequence. Unlike simple feedforward networks (or their iterative equivalents), the errors are not only assessed at the final layer or time. The output units must adopt the appropriate states during the forward iteration, and so during the back-propagation phase, errors are injected at each time-step by comparing the remembered actual states of the oufnut units with their desired states.
Related topics:
AerodynamicsTectonics, erosion and climateLES and DNS
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved