How do you apply regularization techniques, such as L1 or L2 regularization, in PyTorch?

In order to apply regularization techniques, such as L1 or L2 regularization, in PyTorch, we need to modify the loss function of our model during training. These techniques are commonly used to prevent overfitting, which is the phenomenon of a model becoming too specialized to the training data and performing poorly on new, unseen data.

L1 and L2 regularization differ in the type of penalty they impose on the weights of the model. L1 regularization encourages the weights to be sparse by adding the sum of the absolute values of the weights to the loss function, while L2 regularization adds the sum of the squares of the weights to the loss function, thereby encouraging the weights to be small. Let’s see how we can apply these techniques in PyTorch.

## L2 Regularization

To apply L2 regularization in PyTorch, we can modify the loss function of our model as follows:

import torch.nn as nn
import torch.optim as optim

# Define the model architecture
class MyModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

# Initialize the model and the optimizer
model = MyModel(100, 50, 10)
optimizer = optim.SGD(model.parameters(), lr=0.01)

# Define the loss function with L2 regularization
criterion = nn.CrossEntropyLoss()
l2_lambda = 0.01
for param in model.parameters():
    criterion = criterion + l2_lambda * torch.norm(param, 2)

# Train the model
for epoch in range(num_epochs):
    # Forward pass
    outputs = model(inputs)
    loss = criterion(outputs, labels)

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

In the above code, we initialize our model and optimizer, and then define our loss function as the ‘nn.CrossEntropyLoss()‘. We then add L2 regularization penalty to the loss function by iterating through all the model parameters and adding their L2 norm multiplied by ‘l2_lambda‘ to the ‘criterion‘ object.

During training, we use this modified ‘criterion‘ object to compute the loss and perform backpropagation. The optimizer then updates the model weights using the gradients computed during backpropagation.

## L1 Regularization

To apply L1 regularization in PyTorch, we use a similar approach to L2 regularization, but this time adding the absolute values of the weights to the loss function:

# Define the loss function with L1 regularization
criterion = nn.CrossEntropyLoss()
l1_lambda = 0.01
for param in model.parameters():
    criterion = criterion + l1_lambda * torch.norm(param, 1)

The rest of the code remains the same as in the L2 regularization example. It’s important to note that, while both L1 and L2 regularization can help prevent overfitting, they each have different effects on the model weights, and it’s worth experimenting with both to see which one works best for a particular task.