FQGAN
📌 核心贡献
提出 Factorized Quantization（FQ）将大 codebook 分解为多个独立子 codebook，解决 VQGAN 扩大 codebook 时的训练不稳定问题。引入去纠缠正则化（disentanglement regularization）使各子 codebook 捕获互补的视觉特征，并结合 CLIP/DINO 表征学习注入语义信息，显著提升重建质量和 AR 图像生成性能。
📖 摘要
Generative transformers have experienced rapid popularity growth in the computer vision community in synthesizing high-fidelity and high-resolution images. The best generative transformer models so far, however, still treat an image naively as a sequence of tokens, and decode an image sequentially following the raster scan ordering (i.e. line-by-line). We find this strategy neither optimal nor efficient. This paper proposes a novel image synthesis paradigm using a bidirectional transformer decoder, which we term MaskGIT. During training, MaskGIT learns to predict randomly masked tokens by attending to tokens in all directions. At inference time, the model begins with generating all tokens of an image simultaneously, and then refines the image iteratively conditioned on the previous generation. Our experiments demonstrate that MaskGIT significantly outperforms the state-of-the-art transformer model on the ImageNet dataset, and accelerates autoregressive decoding by up to 64x. Besides, we illustrate that MaskGIT can be easily extended to various image editing tasks, such as inpainting, extrapolation, and image manipulation.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | |
| 发表 | 2024-11-25 |
| 分类 | cs.CV |
| 链接 | arXiv |
📝 我的笔记
🔗 相关论文
基于/改进自： VQGAN, FSQ 对比基线： VQGAN, FSQ, ViT-VQGAN 同方向： UniTok, GigaTok, TokenFlow