MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling
📌 核心贡献
MMAR提出了一种新的多模态自回归（MMAR）概率建模框架，旨在解决现有方法在多模态理解任务中因图像离散化或扩散去噪步骤导致图像信息丢失的问题。该方法通过摄入连续值图像令牌来有效避免信息损失，从而实现对图像理解和生成更精确的联合概率建模。
📖 摘要
Recent advancements in multi-modal large language models have propelled the development of joint probabilistic models capable of both image understanding and generation. However, we have identified that recent methods suffer from loss of image information during understanding task, due to either image discretization or diffusion denoising steps. To address this issue, we propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework. Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss in an efficient way. Differing from diffusion-based approaches, we disentangle the diffusion process from auto-regressive backbone model by employing a light-weight diffusion head on top each auto-regressed image patch embedding. In this way, when the model transits from image generation to understanding through text generation, the backbone model’s hidden representation of the image is not limited to the last denoising step. To successfully train our method, we also propose a theoretically proven technique that addresses the numerical stability issue and a training strategy that balances the generation and understanding task goals. Extensive evaluations on 18 image understanding benchmarks show that MMAR significantly outperforms most of the existing joint multi-modal models, surpassing the method that employs pre-trained CLIP vision encoder. Meanwhile, MMAR is able to generate high quality images. We also show that our method is scalable with larger data and model size.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2024-10-14 |
| 分类 | cs.CV |
| 链接 | arXiv |