Academia.eduAcademia.edu

Figure 21 – uploaded by Mark Bishop

See full PDFdownloadDownload figure
FIGURE 21. The two input OR and AND functions are linearly separable

Figure 21 The two input OR and AND functions are linearly separable

Related Figures (22)

Total association: in which there is unrestricted association between previous events and/or arbitrary concepts. E.g. James famously describes ‘The memory of a walk followed by a romantic dinner’. Using these principles James offers the following explanation of three types of associa- tive mental processes involved in spontaneous thought: unrestricted association; partial association and focussed association.
Total association: in which there is unrestricted association between previous events and/or arbitrary concepts. E.g. James famously describes ‘The memory of a walk followed by a romantic dinner’. Using these principles James offers the following explanation of three types of associa- tive mental processes involved in spontaneous thought: unrestricted association; partial association and focussed association.
To recall a forgotten item James envisages a situation where two groups of three memories [a,b,c] and [l,m,n] are fully associated (& interconnected) with the ‘forgotten item’ Z. By the process of total association activation of [a,b,c] will eventually propagate to activate [l,m,n] which together [a,b,c, 1,m,n] will serve to activate the forgotten memory Z, see Figure 2.
To recall a forgotten item James envisages a situation where two groups of three memories [a,b,c] and [l,m,n] are fully associated (& interconnected) with the ‘forgotten item’ Z. By the process of total association activation of [a,b,c] will eventually propagate to activate [l,m,n] which together [a,b,c, 1,m,n] will serve to activate the forgotten memory Z, see Figure 2.
Central to the search to resolve this issue was the ability to stain tissue such that it shows up better under microscopes (e.g. see Figure 3), research pioneered by the Bo- hemian anatomist Johannes Evangelista Purkinnje. Johannes worked at the University of Prague where he successfully made the then thinnest known slices of brain tissue; to make thes slices Purkinnje was amongst the first to use a device called a microtome; a tool to cut extremely thin slices of material.
Central to the search to resolve this issue was the ability to stain tissue such that it shows up better under microscopes (e.g. see Figure 3), research pioneered by the Bo- hemian anatomist Johannes Evangelista Purkinnje. Johannes worked at the University of Prague where he successfully made the then thinnest known slices of brain tissue; to make thes slices Purkinnje was amongst the first to use a device called a microtome; a tool to cut extremely thin slices of material.
Activity at the 4, input to an n input neuron is represented by the symbol X; and the effect of the 274, synapse by a weight W;, hence the net effect of the i4, input on the itn Synapse on the MCP cell is thus X; x W;. Thus the ‘modern’ MCP cell (see Figure 4) is denoted as firing if: NB. In a further modern generalisation of the MCP neuron, the MCP output is defined by an arbitrary function of the weighted sum of its input. This function is called the neuron’s activation function. Example activation functions include: linear summation; the Heaviside (unit step or threshold) function, usually denoted by H, which is a discontinuous function whose value is 0 for negative argument and +1 for positive argument; and the oft deployed sigmoid function (which offers a continuous, differentiable, approximation to the Heaviside function).
Activity at the 4, input to an n input neuron is represented by the symbol X; and the effect of the 274, synapse by a weight W;, hence the net effect of the i4, input on the itn Synapse on the MCP cell is thus X; x W;. Thus the ‘modern’ MCP cell (see Figure 4) is denoted as firing if: NB. In a further modern generalisation of the MCP neuron, the MCP output is defined by an arbitrary function of the weighted sum of its input. This function is called the neuron’s activation function. Example activation functions include: linear summation; the Heaviside (unit step or threshold) function, usually denoted by H, which is a discontinuous function whose value is 0 for negative argument and +1 for positive argument; and the oft deployed sigmoid function (which offers a continuous, differentiable, approximation to the Heaviside function).
FIGURE 5. The link between Connectionism and Associationism
FIGURE 5. The link between Connectionism and Associationism
FIGURE 6. The structure of Rosenblatt’s Perceptron
FIGURE 6. The structure of Rosenblatt’s Perceptron
Building upon Bledsoe and Browning’s earlier insights into n-tuple pattern classifica- tion [23], Aleksander noted that if the inputs and outputs to an MCP cell are constrainec to be binary, the MCP cell computes the ‘Binary Discriminant Function’; the operatior of such a [Binary Discriminant] neuron (see Figure 7) being to classify (discriminate) it: binary input data into one of two sets: TRUE or FALSE. Hence a BDN with n binary (0/1) inputs can classify 2” possible binary patterns. For example, a BDN with thre« input lines, (a, b, c) can perform any binary discrimination between 2° = 8 possible binary input pattern vectors (including functions Minsky and Papert had identified as hard for the single layer perceptron such as XOR and parity).
Building upon Bledsoe and Browning’s earlier insights into n-tuple pattern classifica- tion [23], Aleksander noted that if the inputs and outputs to an MCP cell are constrainec to be binary, the MCP cell computes the ‘Binary Discriminant Function’; the operatior of such a [Binary Discriminant] neuron (see Figure 7) being to classify (discriminate) it: binary input data into one of two sets: TRUE or FALSE. Hence a BDN with n binary (0/1) inputs can classify 2” possible binary patterns. For example, a BDN with thre« input lines, (a, b, c) can perform any binary discrimination between 2° = 8 possible binary input pattern vectors (including functions Minsky and Papert had identified as hard for the single layer perceptron such as XOR and parity).
FIGURE 8. The use of a RAM chip as a Binary Discriminant Neuron
FIGURE 8. The use of a RAM chip as a Binary Discriminant Neuron
FIGURE 9. Generalisation in a Logical (or ‘weightless’) neural network sight patterns that will result in maximum FOUR RAMs firing (the four training images plus new unseen vectors defined by OR-ing together input vectors [2+3], [2+4], [3+4], 2+3-+4]); for a comprehensive review of the capabilities, limitations and extensions of the weightless network paradigm see Ludemir et al [70].
FIGURE 9. Generalisation in a Logical (or ‘weightless’) neural network sight patterns that will result in maximum FOUR RAMs firing (the four training images plus new unseen vectors defined by OR-ing together input vectors [2+3], [2+4], [3+4], 2+3-+4]); for a comprehensive review of the capabilities, limitations and extensions of the weightless network paradigm see Ludemir et al [70].
FIGURE 10. Weights in a binary Hopfield network to learn a binary vector representing ‘Mark’ |-1 -1 +1 +1 -1 +1]
FIGURE 10. Weights in a binary Hopfield network to learn a binary vector representing ‘Mark’ |-1 -1 +1 +1 -1 +1]
If the algorithm cannot find a suitable archetype, it creates a new one by adding an ‘uncommitted neuron’ to the recognition layer. The vigilance parameter has considerable influence on the system: higher vigilance (tending to 1.0) produces highly nuanced ‘memories’ (i.e. many, fine-grained categories), while lower vigilance results (around 0.4 or less) in more general, ‘fuzzy’ memories (i.e. fewer, more generic, categories).
If the algorithm cannot find a suitable archetype, it creates a new one by adding an ‘uncommitted neuron’ to the recognition layer. The vigilance parameter has considerable influence on the system: higher vigilance (tending to 1.0) produces highly nuanced ‘memories’ (i.e. many, fine-grained categories), while lower vigilance results (around 0.4 or less) in more general, ‘fuzzy’ memories (i.e. fewer, more generic, categories).
A Kohonen feature map can be considered as a fully inter-connected network of N MCP style cells (defined by a weight vector n), with in addition each neuron also having a further set of weighted inputs to an M element external input vector (defined by the weight vector m), see Figure 12. Like the ART classifier, the Kohonen network also belongs to the class of unsupervised neural networks. In its original form it was significantly biologically inspired. A Kohonen network consists of an array of adaptable neurons - the outputs of which form a ‘feature map’. Over time each neuron becomes ‘tuned’ to a particular input vector, by adaption of its weight vector. After training, similar input vectors will excite similar regions of this ‘feature map’. f% is typically a sigmoid squashing function whose output is always positive. The n internal weights are clamped as a function of the sinc operator and distance, (see Figure 13).
A Kohonen feature map can be considered as a fully inter-connected network of N MCP style cells (defined by a weight vector n), with in addition each neuron also having a further set of weighted inputs to an M element external input vector (defined by the weight vector m), see Figure 12. Like the ART classifier, the Kohonen network also belongs to the class of unsupervised neural networks. In its original form it was significantly biologically inspired. A Kohonen network consists of an array of adaptable neurons - the outputs of which form a ‘feature map’. Over time each neuron becomes ‘tuned’ to a particular input vector, by adaption of its weight vector. After training, similar input vectors will excite similar regions of this ‘feature map’. f% is typically a sigmoid squashing function whose output is always positive. The n internal weights are clamped as a function of the sinc operator and distance, (see Figure 13).
FIGURE 13. The ‘Mexican hat’ operator, or sinc function, defining inter- nal weight values as a function of distance
FIGURE 13. The ‘Mexican hat’ operator, or sinc function, defining inter- nal weight values as a function of distance
FicurRE 14. An illustration of Kohonen network weight adaption with random pairs of real-valued inputs
FicurRE 14. An illustration of Kohonen network weight adaption with random pairs of real-valued inputs
FIGURE 15. The Radial Basis Function network activation of each hidden layer node by a coefficient and sums the resulting values to produce a network output, see Figure 15. A RBF network is defined by two parameters:
FIGURE 15. The Radial Basis Function network activation of each hidden layer node by a coefficient and sums the resulting values to produce a network output, see Figure 15. A RBF network is defined by two parameters:
FIGURE 17. Can a diameter limited Perceptron compute the ‘connect- edness’ predicate Nievergelt identifies the main focus of the book as an analysis of the computational power of the single layer perception (SLP) and reports its two key conclusions as being: (a) that a [single layer] diameter limited perceptron cannot compute the ‘connectedness’ predicate and (b) that a single layer [fixed order] perceptron cannot compute the k-input ‘parity’ (XOR) problem. 11.2. The ‘connectedness’ predicate. A ‘diameter limited perceptron (DLP) is ¢ perceptron in which the inputs to its A-Units must all fall within a receptive field of siz D.
FIGURE 17. Can a diameter limited Perceptron compute the ‘connect- edness’ predicate Nievergelt identifies the main focus of the book as an analysis of the computational power of the single layer perception (SLP) and reports its two key conclusions as being: (a) that a [single layer] diameter limited perceptron cannot compute the ‘connectedness’ predicate and (b) that a single layer [fixed order] perceptron cannot compute the k-input ‘parity’ (XOR) problem. 11.2. The ‘connectedness’ predicate. A ‘diameter limited perceptron (DLP) is ¢ perceptron in which the inputs to its A-Units must all fall within a receptive field of siz D.
FIGURE 18. The [2 x 2]‘blob’ detecting Perceptron 11.3. The ‘order’ of a perceptron. The order of a perceptron is defined as the largest number of inputs to any of its A-Units; for perceptrons to be useful this ‘order’ must be fixed at some value smaller than the size of the retina. Consider a simple problem: to design a perceptron to recognise (fire) if there is one or more groups of [2 x 2] black pixels on its input retina, (see Figure 18).
FIGURE 18. The [2 x 2]‘blob’ detecting Perceptron 11.3. The ‘order’ of a perceptron. The order of a perceptron is defined as the largest number of inputs to any of its A-Units; for perceptrons to be useful this ‘order’ must be fixed at some value smaller than the size of the retina. Consider a simple problem: to design a perceptron to recognise (fire) if there is one or more groups of [2 x 2] black pixels on its input retina, (see Figure 18).
11.4.1. Can an order (1) perceptron solve the odd parity problem? Consider the 3-input odd parity problem, (see Figure 19). An ‘order (1) perceptron’ is functionally equivalent to asimple MCP cell. To solve the odd parity problem an order (1) perceptron with three A-Units (A;, Ag and A3) must fire - output one - when the number of active elements on the retina is odd. FIGURE 19. The 3-input ‘odd parity’ function The desired perceptron response for a retina of size three is depicted in Figure 19. Thus to depict the function described in Figure 19 the perceptron must learn a set of weights, W,, W2, W3 that satisfy the following constrain:
11.4.1. Can an order (1) perceptron solve the odd parity problem? Consider the 3-input odd parity problem, (see Figure 19). An ‘order (1) perceptron’ is functionally equivalent to asimple MCP cell. To solve the odd parity problem an order (1) perceptron with three A-Units (A;, Ag and A3) must fire - output one - when the number of active elements on the retina is odd. FIGURE 19. The 3-input ‘odd parity’ function The desired perceptron response for a retina of size three is depicted in Figure 19. Thus to depict the function described in Figure 19 the perceptron must learn a set of weights, W,, W2, W3 that satisfy the following constrain:
The firing patterns of the twelve A-Units, for the eight possible input patterns (RO .. R7), are shown in Figure 20; where A, is the weight to A-Unit An. FIGURE 20. Response of A-Unit masks to possible input patterns
The firing patterns of the twelve A-Units, for the eight possible input patterns (RO .. R7), are shown in Figure 20; where A, is the weight to A-Unit An. FIGURE 20. Response of A-Unit masks to possible input patterns
FIGURE 22. The two input XOR function is not linearly separable
FIGURE 22. The two input XOR function is not linearly separable
FIGURE 23. Mapping the two input XOR function into a 3D space
FIGURE 23. Mapping the two input XOR function into a 3D space
Related topics:
PsychologyCognitive ScienceEmbodied Cognition
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved