Backpropagation Explained for Beginners (Part 1): Building the Intuition
to understand backpropagation?
If you’re trying to understand how modern AI systems like large language models (LLMs) are trained, backpropagation is one of the most important concepts to understand.
But if you ask me how I felt when I encountered it, I was completely lost by looking at the math equations. It felt like a mental block for me.
I realized and wanted to start from scratch and build my understanding one step at a time.
That journey began with my previous article, where we built a neural network from scratch using a simple dataset and understood how it makes predictions.
The blog received a great response. Thank you for that!
Now, let’s continue with the same approach. We will break down backpropagation step-by-step, keeping it as simple and intuitive as before.
Before we begin, I just want to say one thing. We’ll take this one step at a time.
Topics like backpropagation can feel overwhelming at first, but once we build a strong foundation, everything else becomes much easier to understand.
So, let’s get started.
Welcome back!
Let’s continue our learning journey through deep learning.
We already have a basic understanding of neural networks, which we explored using a simple dataset in the previous blog.
Now, let’s first recall what we learned in the previous blog on neural networks.
Quick Recap
We considered this simple dataset.
After plotting the data, it looked like this:
We observed that a single line was not enough to fit it. So, we decided to solve it using neural networks.
Next, we got to know about the equation of a single neuron, and after that, we learned about the different layers in a neural network.
For simplicity, we considered one hidden layer with two hidden neurons.
Next, we observed how the two hidden neurons produced two different linear transformations, and then we wanted to combine them in the output layer.
However, we found out that combining two lines produced another line, not the curve that could fit the data.
This is where we realized the significance of activation functions, as they introduce non-linearity into the model.
So, we passed the outputs from the hidden neurons through the activation function (ReLU) and then combined them in the output layer.
In other words, we took the linear combination of the outputs from the activation function in the output layer, and finally, we obtained the curve.
In the previous blog we built the neural network architecture and saw how it makes predictions through forward propagation.
Before we learn how backpropagation works, let’s first look at the values produced at each layer during the forward pass which we discussed in earlier blog.
We’ll use these values throughout the blog to understand how the network learns by updating its parameters.
Why Does the Network Need to Learn?
When we look at the final curve produced by our neural network, we can see that it is not a good fit.
For example, when the hours studied (x) is 1, the actual exam score is 55, but our neural network predicts it as 28, which is a huge difference.
Now, we need to make our neural network perform better, which means it should predict values that are much closer to the actual exam scores.
To do that, the neural network needs to learn. By learning, we mean figuring out which parameters should be increased and which should be decreased to reduce the loss.
Learning from a Familiar Example
Now, how can we do this?
At this point, we don’t know how to do that.
Let’s do one thing. Let’s proceed with what we already know.
But what do we already know?
We already have an idea about simple linear regression, how the loss is calculated, and how the bowl curve looks.
Maybe we can learn something from it.
In simple linear regression, we need to find the optimal values for β0 (intercept) and β1 (slope).
Of course, we already have formulas, but we also derived them ourselves.
What we did was plot a graph with three axes. One axis represented (β0), the second represented (β1), and the third represented the loss.
We plotted the loss values for different (β0) and (β1) values and observed a bowl-shaped curve.
We then understood that the minimum loss occurs at the bottom of the curve, where the slope of the loss surface becomes zero.
To find that point, we used partial differentiation and eventually solved the resulting equations to obtain the formulas.
In simple linear regression, we can use different loss functions such as the Sum of Squared Errors (SSE), Mean Squared Error (MSE), or other suitable loss functions depending on the problem.
Here, we’ll consider the Mean Squared Error (MSE) as our loss function.
For simple linear regression, the loss function is
\[
L(\beta_0,\beta_1)=\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\hat{y}_i\right)^2
\]
where
\[
\hat{y}_i=\beta_0+\beta_1x_i.
\]
Notice that the loss depends only on two parameters, \[\beta_0\] and \[\beta_1\]
We have to find the values of \[\beta_0\] and \[\beta_1\] that minimize this loss.
Now, let’s look at our neural network.
Since our current problem is a non-linear regression problem, we can continue using the same Mean Squared Error (MSE).
The loss function can now be written as
\[
L(w_1,w_2,w_3,w_4,b_1,b_2,b_3)
=
\frac{1}{n}
\sum_{i=1}^{n}
\left(y_i-\hat{y}_i\right)^2.
\]
However, unlike simple linear regression, our prediction is no longer given by
\[
\hat{y}=\beta_0+\beta_1x.
\]
Instead, it is produced by the entire neural network.
For our neural network,
\[
\hat{y}_i
=
w_3\,\mathrm{ReLU}(w_1x_i+b_1)
+
w_4\,\mathrm{ReLU}(w_2x_i+b_2)
+
b_3.
\]
As a result, the loss no longer depends on just two parameters. It now depends on all seven parameters of the neural network, which are \[w_1,w_2,w_3,w_4,b_1,b_2,b_3\]
Just like in simple linear regression, our goal is still the same: find the values of these parameters that minimize the loss.
To achieve that, we need to understand how the loss changes when we change each parameter individually while keeping the remaining parameters fixed.
In other words, we need to compute partial derivatives such as
\[
\frac{\partial L}{\partial w_1},
\quad
\frac{\partial L}{\partial w_2},
\quad
\frac{\partial L}{\partial w_3},
\quad
\ldots,
\quad
\frac{\partial L}{\partial b_3}.
\]
These partial derivatives tell us how sensitive the loss is to each parameter and help us determine whether that parameter should be increased or decreased to reduce the loss.
Setting the Goal
In simple linear regression, when we plot the loss values for different combinations of the slope and intercept, we get a bowl-shaped curve in three-dimensional space.
For our neural network, however, we cannot visualize the loss surface in the same way because it now exists in eight-dimensional space.
Even though we can’t visualize it, our objective remains the same which is to find the parameter values that minimize the loss.
Time to Understand Chain Rule
Now, based on what we already know from simple linear regression, we found a way to proceed further, which is to compute the partial derivatives of the loss with respect to each parameter.
The parameters are
\[w_1,w_2,w_3,w_4,b_1,b_2,b_3\]
But before we proceed, there is one important concept that we need to understand, and that is the chain rule because it is the foundation of everything we’re going to do next.
The chain rule is used whenever one quantity depends on another quantity, which in turn depends on another quantity.
Let’s understand this with a simple example.
Suppose
\[
y=x^2
\]
and
\[
z=y^3
\]
Now, we want to find
\[
\frac{dz}{dx}
\]
First, let’s find this derivative using classical differentiation.
Notice that \[z\] is written in terms of \[y\] not \[x\] Since we want the derivative with respect to \[x\] we can first eliminate the intermediate variable by substituting \[y=x^2\] into the equation for \[z\]
Substituting,
\[
z=(x^2)^3=x^6
\]
Now the expression depends only on \[x\] so we can differentiate it directly.
Using the power rule,
\[
\frac{dz}{dx}
=
\frac{d}{dx}(x^6)
=
6x^5
\]
This method can be easy for simple problems like this one because we can easily substitute one expression into another.
However, imagine a much bigger expression with several intermediate variables.
Rewriting the entire equation before differentiating would become difficult and there is a higher chance for mistakes.
Instead of combining everything into a single expression first, we have a much more systematic approach called the chain rule.
Instead of eliminating the intermediate variables, with the chain rule we can work through them one step at a time.
Let’s see how we can implement chain rule.
We already know that, \[\frac{dz}{dx}\] tells us how much \[z\] changes when we make a very small change in \[x\]
Here \[z\] does not depend directly on \[x\]
Instead, the relationship looks like this:
\[
x \rightarrow y \rightarrow z.
\]
This means that whenever \[x\] changes, it first changes \[y\] and that change in \[y\] then changes \[z\]
Now instead of trying to differentiate everything at once, the chain rule tells us to break the problem into smaller pieces.
\[
\frac{dz}{dx}=\frac{dz}{dy}\times\frac{dy}{dx}
\]
Now let’s calculate each part separately.
Since
\[
z=y^3,
\]
we get
\[
\frac{dz}{dy}=3y^2.
\]
Similarly, since
\[
y=x^2,
\]
we get
\[
\frac{dy}{dx}=2x.
\]
Multiplying these together,
\[
\frac{dz}{dx}=3y^2\times2x.
\]
Finally, we know that
\[
y=x^2,
\]
so we substitute it back into the equation.
\[
\frac{dz}{dx}=3(x^2)^2\times2x=6x^5.
\]
The important thing we can observe here is that we never differentiated the entire expression at a time.
Instead, we broke it into smaller derivatives, solved them one by one, and then multiplied them together.
We’ll use exactly the same idea in our neural network.
The only difference is that the chain is now a little longer.
Solving step by step using Classical Differentiation Method
Now that we understand the chain rule, let’s proceed to calculate the partial derivatives with respect to each parameter.
Until now, we used specific values for the weights and biases to understand how the forward pass works. However, our objective is to learn these values from the data.
So, instead of using fixed values, let’s represent them using parameters first.
The output of our neural network is given by
\[
\hat{y}=w_3a_1+w_4a_2+b_3
\]
where
\[
a_1=\mathrm{ReLU}(z_1)
\]
\[
z_1=w_1x+b_1
\]
and
\[
a_2=\mathrm{ReLU}(z_2)
\]
\[
z_2=w_2x+b_2
\]
Using this output, we can calculate the Mean Squared Error (MSE), which is the loss function of our neural network.
The general MSE equation is
\[
L(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2
\]
Now, let’s substitute the prediction equation into the loss function.
\[
L(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\left(w_3a_{1i}+w_4a_{2i}+b_3\right)\right)^2
\]
Since
\[
a_{1i}=\mathrm{ReLU}(w_1x_i+b_1)
\]
and
\[
a_{2i}=\mathrm{ReLU}(w_2x_i+b_2)
\]
the complete loss function becomes
\[
L(w_1,w_2,w_3,w_4,b_1,b_2,b_3)=\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\left(w_3\mathrm{ReLU}(w_1x_i+b_1)+w_4\mathrm{ReLU}(w_2x_i+b_2)+b_3\right)\right)^2
\]
Now, let’s start finding the partial derivative with respect to any one of the parameters, let’s begin with
\[w_1\]
So, we have to calculate
\[
\frac{\partial}{\partial w_1}\left[\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\left(w_3,\mathrm{ReLU}(w_1x_i+b_1)+w_4,\mathrm{ReLU}(w_2x_i+b_2)+b_3\right)\right)^2\right]
\]
This equation seems difficult to solve. How can we find the partial derivative with respect to
\[w_1\]
from such a large equation?
Let’s proceed using the same ideas from differentiation that we already know.
Instead of differentiating everything at once, we’ll simplify the problem step by step.
We want to compute
\[
\frac{\partial L}{\partial w_1}
\]
Substitute the loss function into the derivative.
\[
\frac{\partial L}{\partial w_1}
=
\frac{\partial}{\partial w_1}
\left(
\frac{1}{n}
\sum_{i=1}^{n}
(y_i-\hat{y}_i)^2
\right)
\]
Notice that we do not substitute the full expression for
\[\hat{y}_i\]
yet. We will do that only when it becomes necessary.
As \[\frac{1}{n}\] is a constant, we know that it can be moved outside the derivative.
\[
\frac{\partial L}{\partial w_1}
=
\frac{1}{n}
\frac{\partial}{\partial w_1}
\left(
\sum_{i=1}^{n}
(y_i-\hat{y}_i)^2
\right)
\]
The summation is also linear, so the derivative can pass through it.
What do we mean by that?
It means in summation we add many terms together and we can differentiate each term separately and then add the derivatives.
\[
\frac{\partial L}{\partial w_1}
=
\frac{1}{n}
\sum_{i=1}^{n}
\frac{\partial}{\partial w_1}
\left(
(y_i-\hat{y}_i)^2
\right)
\]
Now Differentiate the Square
Let
\[
A=y_i-\hat{y}_i
\]
Then
\[
\frac{\partial L}{\partial w_1}
=
\frac{1}{n}
\sum_{i=1}^{n}
\frac{\partial}{\partial w_1}(A^2)
\]
Using the power rule,
\[
\frac{\partial}{\partial w_1}(A^2)
=
2A
\frac{\partial A}{\partial w_1}
\]
Substitute this into the previous equation.
\[
\frac{\partial L}{\partial w_1}
=
\frac{2}{n}
\sum_{i=1}^{n}
A
\frac{\partial A}{\partial w_1}
\]
Replace \[A\] with \[y_i-\hat{y}_i\].
\[
\frac{\partial L}{\partial w_1}
=
\frac{2}{n}
\sum_{i=1}^{n}
(y_i-\hat{y}_i)
\frac{\partial}{\partial w_1}
(y_i-\hat{y}_i)
\]
Differentiate the Expression Inside
We know that the true target (actual observation) \[y_i\] is a constant,
\[
\frac{\partial y_i}{\partial w_1}=0
\]
Therefore,
\[
\frac{\partial}{\partial w_1}
(y_i-\hat{y}_i)
=
-\frac{\partial\hat{y}_i}{\partial w_1}
\]
Substitute this back.
\[
\frac{\partial L}{\partial w_1}
=
-\frac{2}{n}
\sum_{i=1}^{n}
(y_i-\hat{y}_i)
\frac{\partial\hat{y}_i}{\partial w_1}
\]
Now Differentiating the Prediction \[\frac{\partial\hat{y}_i}{\partial w_1}
\]
We know that from the output layer in our neural network
\[
\hat{y}_i
=
w_3a_{1i}
+
w_4a_{2i}
+
b_3
\]
Substitute the equations of hidden neuron activation functions.
\[
\hat{y}_i
=
w_3\mathrm{ReLU}(w_1x_i+b_1)
+
w_4\mathrm{ReLU}(w_2x_i+b_2)
+
b_3
\]
Differentiate with respect to \[w_1\]
\[
\frac{\partial\hat{y}_i}{\partial w_1}
=
\frac{\partial}{\partial w_1}
\left(
w_3\mathrm{ReLU}(w_1x_i+b_1)
+
w_4\mathrm{ReLU}(w_2x_i+b_2)
+
b_3
\right)
\]
Differentiating each term separately.
As \[w_3\] is constant,
\[
\frac{\partial}{\partial w_1}
\left(
w_3\mathrm{ReLU}(w_1x_i+b_1)
\right)
=
w_3
\frac{\partial}{\partial w_1}
\mathrm{ReLU}(w_1x_i+b_1)
\]
The second term contains only \[w_2\] so
\[
\frac{\partial}{\partial w_1}
\left(
w_4\mathrm{ReLU}(w_2x_i+b_2)
\right)
=0
\]
Also,
\[
\frac{\partial b_3}{\partial w_1}=0
\]
Hence,
\[
\frac{\partial\hat{y}_i}{\partial w_1}
=
w_3
\frac{\partial}{\partial w_1}
\mathrm{ReLU}(w_1x_i+b_1)
\]
Differentiate the ReLU Expression
Let
\[
u=w_1x_i+b_1
\]
Then
\[
\mathrm{ReLU}(w_1x_i+b_1)
=
\mathrm{ReLU}(u)
\]
Now Differentiating
\[
u=w_1x_i+b_1
\]
with respect to \[w_1\]
\[
\frac{du}{dw_1}=x_i
\]
Now differentiate the activation.
\[
\frac{d\,\mathrm{ReLU}(u)}{du}
=
\mathrm{ReLU}'(u)
\]
Here we use the chain rule,
\[
\frac{\partial}{\partial w_1}
\mathrm{ReLU}(w_1x_i+b_1)
=
\mathrm{ReLU}'(u)
\frac{du}{dw_1}
\]
Substituting \[\frac{du}{dw_1}=x_i\]
\[
\frac{\partial}{\partial w_1}
\mathrm{ReLU}(w_1x_i+b_1)
=
\mathrm{ReLU}'(u)x_i
\]
Replacing \[u\]
\[
\frac{\partial}{\partial w_1}
\mathrm{ReLU}(w_1x_i+b_1)
=
\mathrm{ReLU}'(w_1x_i+b_1)x_i
\]
This ReLU derivation might get confusing, let’s slow down and see what we actually did here.
We know that the ReLU activation doesn’t depend on directly.
It depends on the value of .
At the same time, the expression depends on .
So when changes, it first changes , which in turn changes the output of the ReLU.
This is exactly the kind of situation where we use the chain rule.
So to find how the ReLU changes with respect to , we first find how changes with respect to , and then how the ReLU changes with respect to .
Finally, we combine these two results using the chain rule.
Now Substitute Back
Earlier, we found
\[
\frac{\partial L}{\partial w_1}
=
-\frac{2}{n}
\sum_{i=1}^{n}
(y_i-\hat{y}_i)
\frac{\partial\hat{y}_i}{\partial w_1}
\]
We also calculated
\[
\frac{\partial\hat{y}_i}{\partial w_1}
=
w_3
\mathrm{ReLU}'(w_1x_i+b_1)
x_i
\]
Substitute this into the previous equation.
\[
\frac{\partial L}{\partial w_1}
=
-\frac{2}{n}
\sum_{i=1}^{n}
(y_i-\hat{y}_i)
w_3
\mathrm{ReLU}'(w_1x_i+b_1)
x_i
\]
Final result
We have derived
\[
\frac{\partial L}{\partial w_1}
=
-\frac{2}{n}
\sum_{i=1}^{n}
(y_i-\hat{y}_i)
w_3
\mathrm{ReLU}'(w_1x_i+b_1)
x_i
\]
This tells us exactly how the loss changes when the weight
\[w_1\]
changes.
At first, this equation may look difficult to understand, but it is actually quite simple.
As it tells us how the total loss changes when we make a very small change to the weight w1, this value is called the gradient, and it is exactly what gradient descent uses to update the weight.
To calculate this gradient, the equation considers every training example in the dataset.
For each training example:
\[(y_i-\hat{y}_i)\] tells us how far the prediction is from the actual value.
\[w_3\] tells us how much the first hidden neuron contributes to the final prediction.
\[\mathrm{ReLU}'(w_1x_i+b_1)\] tells us whether the change in \[w_1\] can pass through the ReLU activation.
\[x_i\] tells us how much a small change in \[w_1\] affects the neuron’s input.
Each training example contributes its own gradient based on these quantities.
We add all of these individual contributions together, and since we are using the Mean Squared Error (MSE) loss function, dividing by gives the average gradient across the entire dataset.
This average gradient tells us how should be adjusted to reduce the overall loss, rather than just the error for a single training example.
Conclusion
If you remember our discussion on Simple Linear Regression, we calculated the partial derivatives with respect to only two parameters.
In this blog, we have successfully derived
\[\frac{\partial L}{\partial w_1}\]
Although the derivation was long, We used the same ideas from calculus that we already knew in every step.
We simply applied differentiation step by step and used the chain rule wherever it was required.
Now, our neural network still has six more parameters, and each of them has its own partial derivative.
So, what do you think?
Do we need to repeat this entire process for every weight and bias?
Fortunately, no.
As neural networks become larger, manually deriving every gradient would quickly become hard and inefficient.
There has to be a better way.
The good news is that we don’t need any new mathematics.
We simply need a better way to organize these calculations.
Everything is still built on the same chain rule we’ve been using throughout this article.
In the next part, we’ll see how the chain rule can be applied efficiently across the entire neural network, leading us to one of the most important algorithms in deep learning: backpropagation.
I hope you learned something from this article. If you’re still confused about neural networks or would like to revisit the basics, you can always read my previous article here.
If you found this helpful, feel free to share it with people who may need it.
In case you have any doubts or thoughts, you can comment on LinkedIn.
“It does not matter how slowly you go as long as you do not stop.”
— Confucius
Thanks for reading, and I’ll see you in Part 2!
