Harmonizing Visual Representations for Unified Multimodal Understanding and Generation
📌 核心贡献
该研究受掩码图像建模（MIM）及其在视觉-语言预训练成功扩展的启发，旨在通过统一的视觉表示，解决在单一多模态框架中统一视觉理解和生成所面临的挑战。该方法旨在克服现有方法中视觉量化（VQ）或变分自编码器（VAE）优先处理内在图像特征而非语义，从而损害理解性能的局限性。
📖 摘要
Unifying visual understanding and generation within a single multimodal framework remains a significant challenge, as the two inherently heterogeneous tasks require representations at different levels of granularity. Current approaches that utilize vector quantization (VQ) or variational autoencoders (VAE) for unified visual representation prioritize intrinsic imagery features over semantics, compromising understanding performance. In this work, we take inspiration from masked image modelling (MIM) that learns rich semantics via a mask-and-reconstruct pre-training and its successful extension to masked autoregressive (MAR) image generation. A preliminary study on the MAR encoder’s representation reveals exceptional linear probing accuracy and precise feature response to visual concepts, which indicates MAR’s potential for visual understanding tasks beyond its original generation role. Based on these insights, we present \emph{Harmon}, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder. Through a three-stage training procedure that progressively optimizes understanding and generation capabilities, Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks while matching the performance of methods with dedicated semantic encoders (e.g., Janus) on image understanding benchmarks. Our code and models will be available at https://github.com/wusize/Harmon.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2025-03-27 |
| 分类 | cs.CV |
| 链接 | arXiv |