Overfit

What happens when a model overfit?

An overfitting model seems to perform really well when you train it. In training it yields very small losses. However, it may perform terribly when evaluating the data.

When does overfitting happen?

Generally speaking, the more data you have, the bigger model you’ll be using. Overfitting happens when you don’t have enough data for your model. Or your model is too big for the task.

How to deal with overfitting?

Add data.

Overfitting happens when data is too few relative to model size. So adding data will help.

Reduce model size.

Reduce the hyper-parameters of the model.

Making the model smaller will definitely help with solving overfitting. However, this is not a practical solution because to reduce the model’s hyper-parameters, the model will have to be re-trained in the process, and that’s a lot of computing power!

Regularization.

p Regularization is a technique to make a model smaller than it is. Regularization makes the model smaller by paralyzing part of the model. For example: dropout layers purposefully drops out part of the model’s inner layer output to make the inner layers appear smaller. L1 and L2 regularization induced penalties on big weights and encourages weights to go to zero so that the inner layer’s effective nodes are reduced (zero stops the data from flowing).