Forward Propagation

by

Forward Propagation

Forward propagation is the process in a neural network to produce an output. It’s called “forward” because the data moves in one direction from input layer to the output layer. The input layer receives the raw data and is passed through the hidden layer. Each neuron in this layer computes the weighted sum of the inputs plus the bias, then an activation function is applied to introduce non-linearity such that the model can understand complex patterns. The result of this layer is passed to the next layer and the process repeats for each layer. The data reaches the output layer after passing through all the hidden layers. Let’s consider an example:



For one input feature x_1 is 0, x_2 is 0 and y is 0. Let’s consider the weights w_1 = 0.5, w_2 = 0.3, w_3 = -0.4, w_4 = 0.8, w_5 = -0.7, w_6 = 0.9 and the bias b_1 = 0.1, b_2 = -0.2, b_3 = 0.2

This is passed through first hidden layer.

Hidden layer one:

In first step weighted sum is calculated

    Weighted sum:

    \[ z = (x_1 \times w_{1}) + (x_2 \times w_{2}) + b_{1} = (0 \times 0.5) + (0 \times 0.3) + 0.1 = 0.1 \]

    Activation (Sigmoid Function):

In step 2, z is passed through the activation function called sigmoid. The main aim of sigmoid curve is to convert the z value between 0 and 1.

    \[ a_{1} = \sigma(z) = \frac{1}{1 + e^{-0.1}} \approx 0.525 \]

Now a_1 passes through the second hidden layer.

Hidden Layer two:

    Weighted sum:

    \[ z_{2} = (x_1 \times w_{3}) + (x_2 \times w_{4}) + b_{2} = (0 \times -0.4) + (0 \times 0.8) + (-0.2) = -0.2 \]

    Activation (Sigmoid function):

    \[ a_{2} = \sigma(z_{2}) = \frac{1}{1 + e^{0.2}} \approx 0.450 \]

Output neuron takes the activations from the hidden layer as input

Output:

    Weighted sum:

    \[ z_o = (a_{1} \times w_{5}) + (a_{2} \times w_{6}) + b_3 = (0.525 \times -0.7) + (0.450 \times 0.9) + 0.2 = -0.3675 + 0.405 + 0.2 \approx 0.2375 \]

    Activation (Sigmoid function):

    \[ a_o = \sigma(z_o) = \frac{1}{1 + e^{-0.2375}} \approx 0.559 \]

Loss function is the error that is the difference between the actual value and predicted value. We need to reduce the error by updating the weights. The process of updating the weight goes backward. This process is called as backward propagation.

Leave a Comment