The ‘Dataset‘ class in PyTorch is a way to represent a collection of data that can be iterated over, and it provides a uniform way of accessing individual records or elements of the dataset. The ‘Dataset‘ class typically handles loading the data from a file or database, and then preprocessing it (e.g. normalizing, scaling, or converting to tensors). The ‘Dataset‘ class is quite flexible and can be used to load various types of data, including images, text, and audio.
The ‘DataLoader‘ class in PyTorch is used to create an iterator over the ‘Dataset‘, which yields batches of data during the training or evaluation process. The ‘DataLoader‘ class allows for convenient batching, shuffling, and parallel loading of the data, which can help reduce training time and improve training accuracy. The ‘DataLoader‘ class also allows for the use of multiprocessing and the creation of worker processes, which can help speed up the loading and preprocessing of the data.
A simple example of how to use ‘Dataset‘ and ‘DataLoader‘ in PyTorch can be:
import torch
from torch.utils.data import Dataset, DataLoader
import numpy as np
# create a custom dataset class
class MyDataset(Dataset):
def __init__(self, data):
self.data = data
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
return torch.from_numpy(self.data[idx])
# create some random data
data = np.random.rand(100, 3)
# create a dataset and dataloader
dataset = MyDataset(data)
dataloader = DataLoader(dataset, batch_size=16, shuffle=True)
# iterate over the dataloader and print each batch
for batch in dataloader:
print(batch.shape)In this example, we create a custom ‘MyDataset‘ class that takes in a numpy array of data, and returns each individual record as a PyTorch tensor, using the ‘__getitem__‘ method. We then use the ‘DataLoader‘ class to create an iterator over the dataset, which yields batches of data with a specified batch size (‘batch_size‘). Finally, we iterate over the ‘DataLoader‘ and print the shape of each batch.