Wasserstein GANs (WGANs) are a variant of Generative Adversarial Networks (GANs) that address some of the challenges associated with training traditional GANs. WGANs replace the traditional GAN’s empirical Jensen-Shannon divergence metric with the Wasserstein distance metric. The key advantage of this is that the Wasserstein distance is well-behaved and easily differentiable almost everywhere, which makes it easier to compute and optimize using gradient-based methods. WGANs also introduce a weight clipping mechanism to enforce a Lipschitz constraint on the discriminator, which stabilizes the training process and leads to improved convergence properties.
In more detail, the key idea behind WGANs is to learn a generator that generates samples which are similar to data samples from a given probability distribution. Like regular GANs, a WGAN consists of two neural networks: a generator G and a discriminator D. In traditional GANs, the generator and discriminator are trained to minimize the Jensen-Shannon divergence between the true data distribution and the generated distribution. However, this objective function may not provide meaningful gradients during the training, leading to unstable and difficult-to-optimize models. Instead of Jensen-Shannon divergence, WGANs minimize the Wasserstein distance between the true data distribution and the generated distribution.
The Wasserstein distance, also known as the earth mover’s distance or Kantorovich-Rubinstein metric, measures how much “work” is needed to transform one probability distribution into another. In the context of GANs, the Wasserstein distance can be interpreted as the expected difference between the predictions of the discriminator on the real vs generated samples. Minimizing the Wasserstein distance encourages the generator to produce samples that are increasingly similar to the data distribution, while simultaneously training the discriminator to distinguish between real and generated samples more accurately.
To ensure that the discriminator has Lipschitz continuity, which is a necessary condition for the existence of the Wasserstein distance, WGANs introduce a weight clipping mechanism on the discriminator. Specifically, the discriminator weights are clipped to a fixed range after each update, so that the maximum absolute value of each weight remains within a pre-determined range. This constraint encourages the discriminator’s gradients to be well-behaved and stable during training, leading to improved convergence properties and avoiding issues such as mode collapse.
In summary, WGANs address some of the challenges associated with training traditional GANs by using the Wasserstein distance metric, which provides meaningful and stable gradients for the generator and discriminator, and by enforcing a Lipschitz constraint on the discriminator using weight clipping. WGANs have shown promising results in a variety of applications such as image generation, text generation, and molecule design.