The recent explosive interest on transformers has suggested their potential to become powerful “universal” models for computer vision tasks,such as classification, detection, and segmentation. However, how further transformers can go- are they ready to take some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs)? Driven by that curiosity, we conduct the first pilot study in building a GAN completely free of convolutions, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed TransGAN, consists of a memory-friendly transformer-based generator that progressively increases feature resolution while decreasing embedding dimension, and a patch-level discriminator that is also transformer-based. We then demonstrate TransGAN to notably benefit from data augmentations (more than standard GANs), a multi-task co-training strategy for the generator, and a locally initialized self-attention that emphasizes the neighborhood smoothness of natural images. Equipped with those findings, TransGAN can effectively scaleup with bigger models and high-resolution image datasets. Our best architecture achieves highly competitive performance compared to current state-of-the-art GANs based on convolutional backbones. Specifically, TransGAN sets newstate-of-the-art IS score of 10.10 and FID score of 25.32 on STL-10. It also reaches competitive8.63 IS score and 11.89 FID score on CIFAR-10, and 12.23 FID score on CelebA64×64, respectively. We also conclude with a discussion of the current limitations and future potential of TransGAN. Read More

#gans

People around the world are using intelligent machines to create new forms of art.

In 2018, in late October, a distinctly odd painting appeared at the fine art auction house Christe’s. At a distance, the painting looks like a 19th-century portrait of an austere gentleman dressed in black. …Our painter is a machine — an intelligent machine. Though the initial estimates had the portrait selling under $10,000, the painting would go on to sell for an incredible $432,500. The portrait was not created by an inspired human mind but was generated by artificial intelligence in the form of Generative Adversarial Networks or GAN. Read More

We introduce the GANsformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range interactions across the image, while maintaining computation of linearly efficiency, that can readily scale to high-resolution synthesis. It iteratively propagates information from a set of latent variables to the evolving visual features and vice versa, to support the refinement of each in light of the other and encourage the emergence of compositional representations of objects and scenes. In contrast to the classic transformer architecture, it utilizes multiplicative integration that allows flexible region-based modulation, and can thus be seen as a generalization of the successful StyleGAN network. We demonstrate the model’s strength and robustness through a careful evaluation over a range of datasets, from simulated multi-object environments to rich real-world indoor and out-door scenes, showing it achieves state-of-the-art results in terms of image quality and diversity, while enjoying fast learning and better data-efficiency. Further qualitative and quantitative experiments offer us an insight into the model’s inner workings, revealing improved interpretability and stronger disentanglement, and illustrating the benefits and efficacy of our approach. Read More

#gans