Cost Function and Backpropagation
- L = total number of layers in the network
- sls = number of units (not counting bias unit) in layer l
- K = number of output units/classes
The (regularized) logistic regression cost function is as follows：
For neural networks, the cost function is a generalization of this equation above, so instead of one output we generate k outputs:
For each training data example, sum of each position in the output vector
This is also called a weight decay term, the lambda value determines the important of the two halves
Back propagation is an algorithm used to minimize the cost function, as it allows us to calculate partial derivatives.
- We want to find parameters Θ which minimize J(Θ):
- Gradient descent
- Advanced optimization algorithms
We want to calculate the partial derivative Ɵ with respect to a single parameter
- Ɵi is the matrix of weights which define the function mapping from layer i to layer i+1
- i here represents the unit in layer l+1 you're mapping to (destination node)
- j is the unit in layer l you're mapping from (origin node)
- l is the layer your mapping from (to layer l+1) (origin layer)
Let's get an idea regarding the intuition of the algorithm
For each node we can calculate (δjl) - this is the error of node j in layer l
But the problem here is, "what is this 'real' value, and how do we calculate it?!"
- The only "real" value we have is our actual classification (our y value) - so that's where we start
To do so, we use the following algorithm:
Set a(1) := x(t)
Perform forward propagation algorithm to compute al for l=2,3,…,L:
Feeding input into the input layer (xi, yi), the sigmoid function applied to the z values gives the activation values
The cost function if there is a single output:
If we consider simple non-multiclass classification (k = 1) and disregard regularization, the cost is computed with:
Intuitively, δj(l) is the "error" for aj(l) (unit j in layer l). More formally, the delta values are actually the derivative of the cost function:
Recall that our derivative is the slope of a line tangent to the cost function, so the steeper the slope the more incorrect we are. Let us consider the following neural network below and see how we could calculate some δj(l) :
Backpropagation in Practice
Implementation Note: Unrolling Parameters
With neural networks, we are working with sets of matrices:
In order to use optimizing functions such as "fminunc()", we will want to "unroll" all the elements and put them into one long vector:
thetaVector = [ Theta1(:); Theta2(:); Theta3(:); ]
If the dimensions of Theta1 is 10x11, Theta2 is 10x11 and Theta3 is 1x11, then we can get back our original matrices from the "unrolled" versions as follows:
Theta1 = reshape(thetaVector(1:110),10,11)
Backpropagation has a lot of details, small bugs can be present and ruin it :-(
Gradient checking will assure that our backpropagation works as intended. We can approximate the derivative of our cost function with:
With multiple theta matrices, we can approximate the derivative with respect to Θj as follows:
A small value for ϵ(epsilon) such as ϵ=10−4, guarantees that the math works out properly. If the value for ϵ* is too small, we can end up with numerical problems.
Hence, we are only adding or subtracting epsilon to the Θj matrix. In octave we can do it as follows:
epsilon = 1e-4;
Once you have verified once that your backpropagation algorithm is correct, you don't need to compute gradApprox again. The code to compute gradApprox can be very slow.
Initializing all theta weights to zero does not work with neural networks. When we backpropagate, all nodes will update to the same value repeatedly. Instead we can randomly initialize our weights for our Θ matrices using the following method:
%If the dimensions of Theta1 is 10x11, Theta2 is 10x11 and Theta3 is 1x11.
Putting It Together
First, pick a network architecture; choose the layout of your neural network, including how many hidden units in each layer and how many layers in total you want to have.
- Number of input units = dimension of features x(i)
- Number of output units = number of classes
- Number of hidden units per layer = usually more the better (must balance with cost of computation as it increases with more hidden units)
- Defaults: 1 hidden layer. If you have more than 1 hidden layer, then it is recommended that you have the same number of units in every hidden layer.
Training a Neural Network
- Randomly initialize the weights
- Implement forward propagation to get hΘ(x(i)) for any x(i)
- Implement the cost function
- Implement backpropagation to compute partial derivatives
- Use gradient checking to confirm that your backpropagation works. Then disable gradient checking.
- Use gradient descent or a built-in optimization function to minimize the cost function with the weights in theta.
When we perform forward and back propagation, we loop on every training example:
for i = 1:m,
The following image gives us an intuition of what is happening as we are implementing our neural network:
Ideally, you want hΘ(x(i)) ≈ y(i). This will minimize our cost function. However, keep in mind that J(Θ) is not convex and thus we can end up in a local minimum instead.