---

Counter Propagation Networks

---

Counterpropagation network

Counterpropagation network (CPN) were proposed by Hecht Nielsen in 1987.They are multilayer network based on the combinations of the input, output, and clustering layers. The application of counterpropagation net are data compression, function approximation and pattern association. The ccounterpropagation network is basically constructed from an instar-outstar model. This model is three layer neural network that performs input-output data mapping, producing an output vector y in response to input vector x, on the basis of competitive learning. The three layer in an instar-outstar model are the input layer, the hidden(competitive) layer and the output layer.

There are two stages involved in the training process of a counterpropagation net. The input vector are clustered in the first stage. In the second stage of training, the weights from the cluster layer units to the output units are tuned to obtain the desired response.

There are two types of counterpropagation network:

1. Full counterpropagation network
2. Forward-only counterpropagation network

Full counterpropagation network

Full CPN efficiently represents a large number of vector pair x:y by adaptively constructing a look-up-table. The full CPN works best if the inverse function exists and is continuous. The vector x and y propagate through the network in a counterflow manner to yield output vector x* and y*.

Architecture of Full Counterpropagation Network

The four major components of the instar-outstar model are the input layer, the instar, the competitive layer and the outstar. For each node in the input layer there is an input value xi. All the instar are grouped into a layer called the competitive layer. Each of the instar responds maximally to a group of input vectors in a different region of space. An outstar model is found to have all the nodes in the output layer and a single node in the competitive layer. The outstar looks like the fan-out of a node.

Training Algorithm for Full Counterpropagation Network:

Step 0: Set the initial weights and the initial learning rare.

Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training.

Step 2: For each of the training input vector pair x: y presented, perform Steps 3-5.

Step 3: Make the X-input layer activations to vector X. Make the Y-inpur layer activations to vector Y.

Step 4: Find the winning cluster unit. If dot product method is used, find rhe cluster unit Zj with target net input: for j = 1 to p.

$$Z_{inj}=\sum _{i=1}^n{x}_i{v}_{ij}+\sum _{k=1}^m{y}_k{w}_{kj}$$

If Euclidean distance method is used, find the cluster unit Z_j whose squared distance from input vectors is the smallest

$$D(j)=\sum _{i=1}^n(x_i-<!--\; -\; -->v_{ij}{)}^2+\sum _{k=1}^m(y_k-<!--\; -\; -->w_{kj}{)}^2$$

If there occurs a tie in case of selection of winner unit, the unit with the smallest index is the winner. Take the winner unit index as J.

Step 5: Update the weights over the calculated winner unit Z_j

$$v_{ij}(new)=v_{ij}(old)+\alpha [x_i-v_{ij}(old)]i=1ton$$

$$w_{kj}(new)=w_{kj}(old)+\beta [y_k-w_{kj}(old)]k=1tom$$

Step 6: Reduce the learning rates α and β

$$\alpha (t+1)=0.5\alpha t$$

$$\beta (t+1)=0.5\beta t$$

Step 7: Test stopping condition for phase-I training.

Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training.

Step 9: Perform Steps 10-13 for each training input pair x:y. Here α and β are small constant values.

Step 10: Make the X-input layer activations to vector x. Make the Y-input layer activations to vector y.

Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J.

Step 12: Update the weights entering into unit Z_J

$$v_{ij}(new)=v_{ij}(old)+\alpha [x_i-v_{ij}(old)]i=1ton$$

$$w_{kj}(new)=w_{kj}(old)+\beta [y_k-w_{kj}(old)]k=1tom$$

Step 13: Update the weights from unit Z_j to the output layers.

$$t_{ji}(new)=t_{ji}(old)+b[x_i-t_{ji}(old)]i=1ton$$

$$u_{jk}(new)=u_{jk}(old)+a[y_k-u_{jk}(old)]k=1tom$$

Step 14: Reduce the learning rates a and b.

$$a(t+1)=0.5at$$

$$b(t+1)=0.5bt$$

Step 15: Test stopping condition for phase-II training.

Forward-only Counterpropagation network:

A simplified version of full CPN is the forward-only CPN. Forward-only CPN uses only the x vector to form the cluster on the Kohonen units during phase I training. In case of forward-only CPN, first input vectors are presented to the input units. First, the weights between the input layer and cluster layer are trained. Then the weights between the cluster layer and output layer are trained. This is a specific competitive network, with target known.

Architecture of forward-only CPN

It consists of three layers: input layer, cluster layer and output layer. Its architecture resembles the back-propagation network, but in CPN there exists interconnections between the units in the cluster layer.

Training Algorithm for Forward-only Counterpropagation network:

Step 0: Initial the weights and learning rare.

Step 1: Perform Steps 2-7 if stopping condition is false for phase-I training.

Step 2: Perform Steps 3-5 for each of uaining input X

Step 3: Set the X-input layer activations to vector X.

Step 4: Compute the winning cluster unit (J). If dot product method is used, find the cluster unit z_j with the largest net input.

$$Z_{inj}=\sum _{i=1}^n{x}_i{v}_{ij}$$

If Euclidean distance method is used, find the cluster unit Z_j whose squared distance from input patterns is the smallest

$$D(j)=\sum _{i=1}^n(x_i-<!--\; -\; -->v_{ij}{)}^2$$

If there exists a tie in the selection of wiriner unit, the unit with the smallest index is chosen as the winner.

Step 5: Perform weight updation for unit Z_j. For i= 1 to n,

$$v_{ij}(new)=v_{ij}(old)+\alpha [x_i-v_{ij}(old)]i=1ton$$

Step 6: Reduce the learning rates α

$$\alpha (t+1)=0.5\alpha t$$

Step 7: Test stopping condition for phase-I training.

Step 8: Perform Steps 9-15 when stopping condition is false for phase-II training.

Step 9: Perform Steps 10-13 for each training input Pair x:y..

Step 10: Set X-input layer activations to vector X. Sec Y-outpur layer activations to vector Y.

Step 11: Find the winning cluster unit (use formulas from Step 4). Take the winner unit index as J.

Step 12: Update the weights entering into unit Z_J,

$$v_{ij}(new)=v_{ij}(old)+\alpha [x_i-v_{ij}(old)]i=1ton$$

Step 13: Update the weights from unit Z_j to the output layers.

$$w_{kj}(new)=w_{kj}(old)+\beta [y_k-w_{kj}(old)]k=1tom$$

Step 14: Reduce the learning rates β.

$$\beta (t+1)=0.5\beta t$$

Step 15: Test stopping condition for phase-II training.