Regularization Techniques used in Neural Networks in Deep Learning

Ideally, the neural networks should never underfit and overfit and maintain good generalization capabilities. For this purpose, we use various regularization techniques in our neural networks. Below is the list of some of the regularization techniques which are commonly used to improve the performance and accuracy of the neural networks in deep learning.

1. L1 and L2 Regularization

L1 and L2 are the most common types of regularization techniques used in machine learning as well as in deep learning algorithms. These update the general cost function by adding another term known as the regularization penalty. 

For more details, please go through my this article.

2. Dropout

Dropout can be seen as temporarily deactivating or ignoring neurons in the hidden layers of a network. Probabilistically dropping out nodes in the network is a simple and effective regularization method. We can switch off some neurons in a layer so that they do not contribute any information or learn any information and the responsibility falls on other active neurons to learn harder and reduce the error.

For more details on dropout, please consider visiting my this post.

3. Data Augmentation

Creating new data by making reasonable modifications to the existing data is called data augmentation. Lets take an example of our MNIST dataset (hand written digits). We can easily generate thousands of new similar images by rotating, flipping, scaling, shifting, zooming in and out, cropping, changing or varying the color of the existing images. 

We can use data augmentation technique when our model is overfitting due to less data.

In many cases in deep learning, increasing the amount of data is not a difficult task as we discussed above the case of MNIST dataset. In machine learning, this task is not that easy as we need labeled data which is not easily available. 

4. Early Stopping

While training a neural network, there will be a point during training when the model will stop generalizing and start learning the noise in the training dataset. This leads to overfitting.

One approach to solve this problem is to treat the number of training epochs as a hyperparameter and train the model multiple times with different values, then select the number of epochs that result in the best performance. 

The downside of this approach is that it requires multiple models to be trained and discarded. This can be computationally inefficient and time-consuming.

Another approach is early stopping. The model is evaluated on a validation dataset after each epoch. If the performance of the model on the validation dataset starts to degrade (e.g. loss begins to increase or accuracy begins to decrease), then the training process is stopped. The model at the time when the training is stopped, is then used and is known to have good generalization performance.

What is Dropout? How does it prevent overfitting in a neural network?

Dropout is an effective regularization technique used in neural networks which increases generalization capabilities of a deep learning model and prevent it from overfitting.

Overfitting in neural networks

Large neural networks trained on relatively small datasets can overfit the training data. Over-fitted neural networks results in poor performance when the model is evaluated on new data. Dropout is an efficient solution to handle this over-fitting problem in neural networks.

What happens in dropout?

Dropout can be seen as temporarily deactivating or ignoring neurons in the hidden layers of a network. Probabilistically dropping out nodes in the network is a simple and effective regularization method. We can switch off some neurons in a layer so that they do not contribute any information or learn any information and the responsibility falls on other active neurons to learn harder and reduce the error.

Points to note about dropout

1. Dropout is implemented per-layer in a neural network. Dropout can be implemented in hidden and input layers, but not in output layers. 

We can use different probabilities for dropout on each layer. As mentioned previously, dropout should not be implemented on output layer, so the output layer would always have keep_prob = 1 and the input layer has high keep_prob such as 0.9 or 1. 

If a hidden layer has keep_prob = 0.8, this means that on each iteration, each unit has 80% probability of being included and 20% probability of being dropped out.

This probability acts as a hyper-parameter and we should carefully decide how many neurons we want to deactivate in a given hidden layer.

2. Dropout can be used with many types of layers, such as dense fully connected layers, convolutional layers, and recurrent layers such as the long short-term memory network (LSTM) layers.

3. Dropout should be implemented only during training phase, not in testing phase. 

4. Dropout can be compared to bagging technique in machine learning. In bagging, all trees are not trained on all the features. Similarly, using dropout, all the hidden layers are not trained on all the features.

Advantages of dropout

1. Reduces overfitting and hence increases the accuracy of the model

2. Improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark datasets.

3. Computationally cheap as compared to other regularization methods.

Disadvantages of dropout

1. Introduces sparsity: If we use dropout to a large extent, activations inside the hidden layers may become sparse. You can correlate it with sparse autoencoders.

2. Dropout makes training process noisy as it forces nodes within a layer to probabilistically take on more or less responsibility for the inputs.

Difference between Ridge Regression (L2 Regularization) and Lasso Regression (L1 Regularization)

Regularization is mainly used to solve the overfitting problem in Machine Learning algorithms and helps in generalizing the prediction ability of ML algorithms. 

If a model is simple, it may be the case that it is not exposed to the significant amount of training data and it may underfit. This model will not be able to generalize the data. 

A complex model can also capture the noisy data which is totally irrelevant to our predictions. This model may perform well in the training data but will not perform well in test data due to overfitting. 

We need to choose the right model in between the simple and the complex model. Regularization helps to choose the preferred model complexity, so that model does not overfit and is better at generalization. 

Regularization is of 3 types:

1. Ridge Regression (L2 Regularization)
2. Lasso Regression (L1 Regularizaion)
3. Elastic Net Regreesion 

Regularization adds some amount of bias (called Regularization Penalty) to the objective function and in return the algorithm gets significant drop in the variance. 

For example, Linear Regression tries to minimize the Loss Function (lets say Sum of the Squared Errors) to get the best fit line. In order to prevent this model from overfitting, we can add Regularization Penalty to the Loss Function. Now the model has to minimize both the Loss Function and the Regularization Penalty. 

The severity of the penalty is found by cross validation. In this way, the final model will never overfit. The severity of the penalty can vary from 0 to positive infinity. If severity is zero, it means we are not considering the regularization at all in our model.

Difference between Ridge Regression (L2 Regularization) and Lasso Regression (L1 Regularization)

1. In L1 regularization, we penalize the absolute value of the weights while in L2 regularization, we penalize the squared value of the weights.

2. In L1 regularization, we can shrink the parameters to zero while in L2 regularization, we can shrink the parameters to as small as possible but not to zero. So, L1 can simply discard the useless features in the dataset and make it simple.

When to use what?

There is no any hard and fast rule. If you need to eliminate some useless features from the dataset, L1 should be preferred. But, if you cannot afford to eliminate any feature from your dataset, use L2. In fact we should try both L1 and L2 regularization and check which results in better generalization. We can also use Elastic Net Regression which combines the features of both L1 and L2 regularization.

What is the cause of Overfitting and how can we avoid it in Machine Learning?

How to avoid overfitting in Machine Learning? What are the various ways to deal with the overfitting of the data in Machine Learning? This is the very important question to consider in the world of Data Science and Machine Learning. 

Lets start this discussion with a small story of two brothers: Tom and Harry.

Tom and Harry study in First standard. Tomorrow is their Maths exam. Syllabus contains 20 questions (5 questions each from Addition, Subtraction, Multiplication and Division questions).

5 Addition questions in the syllabus are:

2+5=7
3+2=5
1+8=9
4+2=6
7+1=8

Tom learnt only Addition and Subtraction.
Harry memorized all the 20 questions.

Exam Result: Tom just managed to pass while Harry became one of the toppers.

Next day, Harry was given following Addition problems to solve:

3+2 (Harry answered 5 because this was in syllabus and he had memorized it)
1+1 (Harry failed to answer this simple question because this was not in the syllabus)
2+1 (Harry failed to answer this simple question because this was not in the syllabus)

Tom was able to solve above all but was not able to solve Multiplication and Division problems.

Story finished. Now lets relate it to Underfitting and Overfitting concepts in Machine Learning. 

Consider Maths syllabus as a training dataset.

Tom learnt only small portion of the syllabus. This is Underfitting. Algorithm is not aware of all the scenarios, so will not be able to predict those scenarios in the test dataset as it is not trained in on those scenarios in the training dataset. But it will be able to generalize the scenarios which it learnt.

Harry memorized all the 20 questions. This is Overfitting. Algorithm will not be able to generalize the data in the real environment and will result in high variance.

So, following points should be noted down regarding Overfitting and Underfitting:

1. A good algorithm should have low/reasonable bias in the training dataset. Then it will also have low variance in test dataset which is good sign of a consistent algorithm.

2. If an algorithm overfits in the training dataset (have zero bias), there is a lot of possibility that it will have high variance in the test dataset which is a bad.

3. When a model tries to overfit, it loses its generalization capacity, due to which its shows poor performance in the test dataset.

4. The model which tries to overfit the training set, mainly becomes too complex.

5. The model which tries to underfit the training set, mainly becomes too simple.

What causes Overfitting? 

1. Small training dataset
2. Large number of features in a dataset
3. Noise in the dataset

How to avoid Overfitting? 

We need to find out a way in-between of overfit and underfit. This is achieved by following techniques:

1. Dimensionality Reduction

2. Regularization Technique (Ridge Regression - L2, Lasso Regression - L1, Elastic Net Regression)

3. Cross Validation Techniques

4. Ensemble Learning Techniques (Bagging and Boosting)

• Bagging: Random Forest
• Boosting: AdaBoost, Gradient Boosting Machine (GBM), XGBoost 

I will discuss all these techniques in detail in my next post.