VQGAN
📌 核心贡献
将 CNN 的归纳偏置与 Transformer 的表达能力结合：用 CNN 学习上下文丰富的视觉词汇（VQGAN），再用 Transformer 建模高分辨率图像中 token 的组合关系，实现了首个语义引导的百万像素级图像生成。
📖 摘要
Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet. Code and pretrained models can be found at https://github.com/CompVis/taming-transformers.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | Heidelberg University |
| 发表 | 2020-12-17 |
| 分类 | cs.CV |
| 链接 | arXiv |
📝 我的笔记
🔗 相关论文
基于/改进自： VQ-VAE 对比基线： VQ-VAE, VQ-VAE-2 同方向： ViT-VQGAN, VQGAN-f16, VQGAN-100k