---

Back Propagation

---

Back Propagation Neural Networks

Back Propagation Neural (BPN) was first introduced in 1960s, It is a multilayer neural network consisting of the input layer, at least one hidden layer and output layer. It is a multi-layer feed forward neural network using Gradient descent approach which exploits the chain rule to optimize the parameter. The error which is calculated at the output layer, by comparing the target output and the actual output, will be propagated back towards the input layer.

The main features of Backpropagation are the iterative, recursive and efficient method through which it calculates the updated weight to improve the network until it is not able to perform the task for which it is being trained.

Back Propagation Neural Networks Training Algorithm

Back Propagation Neural will use binary sigmoid activation function. The training will have the following three phases.

• Phase 1 − Feed Forward Phase
• Phase 2 − Back Propagation of error
• Phase 3 − Updating of weights

Step 0 − initialize the weights and the bias(For easy calculation and simplicity, take some small random values but not zero). also initialize the learning rate α(0, α, 1).

Step 1 − Continue step 2-10 when the stopping condition is not true.

Step 2 − Continue step 3-9 for every training pair.

Phase 1

Step 3 − Each input unit receives input signal xi and sends it to the hidden unit for all i = 1 to n

Step 4Calculate the net input at the hidden unit using the following relation − $$Q_{inj}=\underset{i=1}{\overset{n}{\sum <!--\; \sum \; -->}}{x}_i{v}_{ij}+b_{\mathrm{j}}j=1top$$

Now calculate the net output by applying the following sigmoidal activation function $$Q_j=f(Q_{inj})$$

Step 5 Calculate the net input at the output layer unit using the following relation $$y_{ink}=\underset{j=1}{\overset{p}{\sum <!--\; \sum \; -->}}{Q}_j{w}_{jk}+b_{\mathrm{k}}k=1tom$$

Calculate the net output by applying the following sigmoidal activation function $$y_k=f(y_{ink})$$

Phase 2

Step 6 − Compute the error correcting term, in correspondence with the target pattern received at each output unit, as follows $${\mathrm{\delta <!--\; \delta \; -->}}_k=(t_k-<!--\; -\; -->y_k)f^{{}^{\prime }}(y_{ink})$$

The derivative $f^{{}^{\prime }}(y_{ink})$ can be calculated as:

if Binary sigmoid function:$$f^{{}^{\prime }}(y_{ink})=f^{}(y_{ink})(1_{}-<!--\; -\; -->f^{}(y_{ink}))$$

if Bipolar sigmoid function:$$f^{{}^{\prime }}(y_{ink})=(1_{}+f^{}(y_{ink}))(1_{}-<!--\; -\; -->f^{}(y_{ink}))$$

Then, send ${\mathrm{\delta <!--\; \delta \; -->}}_k$ back to the hidden layer

Step 7 − Now each hidden unit will be the sum of its delta inputs from the output units. $${\mathrm{\delta <!--\; \delta \; -->}}_{inj}={\displaystyle \underset{k=1}{\overset{m}{\sum <!--\; \sum \; -->}}{\mathrm{\delta <!--\; \delta \; -->}}_k{w}_{jk}}$$

Error term can be calculated as follows − $${\mathrm{\delta <!--\; \delta \; -->}}_j={\mathrm{\delta <!--\; \delta \; -->}}_{inj}{f}^{{}^{\prime }}(Q_{inj})$$

Phase 3

Step 8 − Each output unit y_k (k = 1 to m) updates the weight and bias as follows $$w_{jk}(new)=w_{jk}(old)+\mathrm{\Delta <!--\; \Delta \; -->}{w}_{jk}$$ $$b_{\mathrm{k}}(new)=b_{\mathrm{k}}(old)+\mathrm{\Delta <!--\; \Delta \; -->}{b}_{\mathrm{k}}$$

$$\mathrm{\Delta <!--\; \Delta \; -->}{w}_{jk}=\mathrm{\alpha <!--\; \alpha \; -->}{\mathrm{\delta <!--\; \delta \; -->}}_k{Q}_j$$ $$\mathrm{\Delta <!--\; \Delta \; -->}{b}_{0k}=\mathrm{\alpha <!--\; \alpha \; -->}{\mathrm{\delta <!--\; \delta \; -->}}_k$$

Step 9 − Each Hidden unit q_j (j = 1 to p) updates the weight and bias as follows $$v_{ij}(new)=v_{ij}(old)+\mathrm{\Delta <!--\; \Delta \; -->}{v}_{ij}$$ $$b_{\mathrm{j}}(new)=b_{\mathrm{j}}(old)+\mathrm{\Delta <!--\; \Delta \; -->}{b}_{\mathrm{j}}$$

$$\mathrm{\Delta <!--\; \Delta \; -->}{v}_{ij}=\mathrm{\alpha <!--\; \alpha \; -->}{\mathrm{\delta <!--\; \delta \; -->}}_j{x}_i$$ $$\mathrm{\Delta <!--\; \Delta \; -->}{b}_{\mathrm{j}}=\mathrm{\alpha <!--\; \alpha \; -->}{\mathrm{\delta <!--\; \delta \; -->}}_j$$

Step 10 − Check for the stopping condition, which may be either the number of epochs reached or the target output matches the actual output.

---

Next Topic :Associate Memory Network