How do you perform model pruning in PyTorch to reduce the size of a trained model while maintaining its accuracy?

Model pruning is a technique used to reduce the size of a deep neural network by removing unnecessary or redundant parameters while preserving the model’s predictive performance. In PyTorch, model pruning can be implemented using several methods such as weight pruning, neuron pruning, and layer pruning.

Here are the steps to perform weight pruning in PyTorch:

1. Train the model using your training dataset and save the trained model weights.

2. Load the saved model weights into a new model instance.

3. Define a pruning criterion that determines which model parameters to prune. PyTorch provides several pruning criteria such as L1 and L2 norm based pruning.

4. Create a pruning scheduler that determines the pruning rate at different iterations. The pruning scheduler can be constant or can vary over time.

5. Wrap the model with the pruning criterion and scheduler using the ‘torch.nn.utils.prune‘ module.

6. Prune the model parameters using the ‘prune‘ method from the ‘torch.nn.utils.prune‘ module at every iteration.

7. Retrain the pruned model using the training dataset until the desired accuracy is achieved.

Here is an example of how to perform weight pruning on a simple neural network in PyTorch:

import torch
import torch.nn as nn
import torch.nn.utils.prune as prune

# Define a neural network
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(768, 512)
        self.fc2 = nn.Linear(512, 10)
        
    def forward(self, x):
        x = self.fc1(x)
        x = nn.functional.relu(x)
        x = self.fc2(x)
        return nn.functional.log_softmax(x, dim=1)

# Define the training dataset
train_data = torch.randn(1000, 768)
train_targets = torch.randint(10, (1000,))

# Train the model and save the weights
model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(10):
    optimizer.zero_grad()
    outputs = model(train_data)
    loss = loss_fn(outputs, train_targets)
    loss.backward()
    optimizer.step()

torch.save(model.state_dict(), 'model_weights.pth')

# Load the saved model weights into a new model instance
pruned_model = Net()
pruned_model.load_state_dict(torch.load('model_weights.pth'))

# Define the pruning criterion and scheduler
pruning_rate = 0.2
pruning_criterion = prune.L1Unstructured
pruning_scheduler = prune.PolynomialDecayPruningSchedule(
    final_sparsity=pruning_rate,
    initial_sparsity=0.1,
    schedule_length=10,
    mode='linear'
)

# Wrap the model with the pruning criterion and scheduler
pruned_model.fc1 = prune.prune(pruned_model.fc1, pruning_criterion, pruning_scheduler)

# Prune the model parameters at every iteration
optimizer = torch.optim.SGD(pruned_model.parameters(), lr=0.01)
for epoch in range(10):
    optimizer.zero_grad()
    outputs = pruned_model(train_data)
    loss = loss_fn(outputs, train_targets)
    loss.backward()
    optimizer.step()
    pruning_scheduler.step()  # Update the pruning rate

# Test the pruned model accuracy
test_data = torch.randn(100, 768)
test_targets = torch.randint(10, (100,))
pruned_model.eval()
with torch.no_grad():
    outputs = pruned_model(test_data)
_, predicted = torch.max(outputs.data, 1)
accuracy = (predicted == test_targets).sum().item() / len(test_targets)
print(f'Accuracy: {accuracy}')  # should be close to the original model's accuracy

In this example, we first define a simple neural network and train it on a dataset, saving the network’s weights.

We then load the saved weights into a new network instance and define a pruning criterion and scheduler. We use the L1 criterion to determine which weights should be pruned and a polynomial decay scheduler to gradually increase the pruning rate over 10 iterations.

Next, we wrap the network’s first fully-connected layer (fc1) with the pruning criterion and schedule using the ‘prune‘ method from ‘torch.nn.utils.prune‘ and then train the pruned model on the same dataset. At each iteration, we call the ‘step‘ method of the pruning scheduler to update the pruning rate.

Finally, we evaluate the accuracy of the pruned model on a test dataset and observe that it is close to the original model’s accuracy. If necessary, this process can be repeated multiple times with higher pruning rates to achieve even smaller models while preserving performance.