[YixuanBan Notes]Image-to-Image Translation with Conditional Adversarial Networks

2017-10-10

Posted by 班怡璇

Abstract:

  1. The conditional adversarial networks can deal with the image-to-image translation problems. And it not only learn the mapping from input image to output image, but also learn a loss function to train this mapping, avoiding hand-engineering loss functions.
  2. This approach can effectively synthesize photos from label maps, reconstruct objects from edge maps, and colorize images, among other tasks.

Introduction:

  1. The traditional image-to-image translation problems need separate, special-purpose machinery, so in this paper, the writer want to propose a common framework for all the problems.
  2. The CNN can be applied in many prediction problems, however, we should tell the CNN what we wish it to minimize (tell the loss function), which is can not be naive, so it could result in blurry.
  3. If the loss function can be learned automatically, the network is called GAN. It tries to learn a loss that tries to classify if the result is real or fake and minimize this loss. In this way, the blurry images can not be tolerated.(learning generative model of data)
  4. The writer’s contribution: 1>demonstrate that on a wide variety of problems, cGANs can produce reasonable results. 2>present a framework sufficient to achieve good results and analyze the effects.

Related works:

  1. The traditional classifications treat the pixels of input images as independent. However, the cGANs treat it as structured.
  2. The proposed GAN is not tailored. It can be used in general.

Method:

  1. Traditional GANs learn a mapping from random noise Z to output image Y. However, the cGANs learn from observed image X and noise Z to image Y.
  2. The G-generator and D-discrimintor are adversarial.
  3. The objective is as below:1111111final obj
  4. Without z , the net could still learn a mapping from x  to y, but would produce deterministic outputs.In initial experiments, the generator simply learned to ignore the noise. So, in the final models, the writer provide noise only in the form of dropout, which result in stochasticity in the output of the nets.
  5. Previous work uses the encoder-decoder network, which the input can pass through a series of layers that progressively downsample, until a bottleneck layer, at which point the process is reversed. However, there is always some information can not be downsampled, the writer add skip connections between each layer i and layer n-i as below.(Add details)architecture
  6. L2 and L1 loss will result on blurry results on image generation problems. Despite this, the accurate low frequency prediction is satisfying. So we do not need an entirely new framework to enforce correctness at the low frequencies.( L1 is OK)
  7. The GAN only tries to model high-frequency structure (L1 tries to model low frequency), so restrict user’s attention to the local image patches is sufficient, besides, this kind of Patch-GAN has fewer parameters, runs faster and can be applied widely.

Experiment:

  1. The writer applies the method to a variety of tasks and datasets as below: 1>Semantic labels – photo, trained on the Cityscapes dataset.  2>Architectural labels – photo, trained on the CMP Facades dataset. 3>Map aerial – photo, trained on data scraped from Google Maps. 4> BW – color photos. 5>Edges – photo. 6>Sketch – photo. 7>Day – night
  2. The writer compares the L1, GAN, cGAN, GAN+L1 and cGAN+L1 methods(objective function), and gives a quantitative and qualitative result of pixel accuracy and class accuracy. Besides, the writer compares the colorfulness of each method.
  3. The writer compares the generator architecture(U-net and encoder-decoder structure). It turns out that the U-net structure has clearer edges.
  4. The writer compares the size of patch both on qualitative and quantitative. From PixelGAN to PatchGans to ImageGANs.

Ref:

  1. https://www.leiphone.com/news/201706/ty7H504cn7l6EVLd.html