PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
📌 核心贡献
PUMA致力于通过多粒度视觉生成能力，赋能统一的多模态大语言模型（MLLM）。该研究旨在解决现有工作在统一MLLM范式下未能充分处理不同图像生成任务（从文本到图像生成的多样性到图像操作的精确可控性）对不同粒度需求的问题。
📖 摘要
Recent advancements in multimodal foundation models have yielded significant progress in vision-language understanding. Initial attempts have also explored the potential of multimodal large language models (MLLMs) for visual content generation. However, existing works have insufficiently addressed the varying granularity demands of different image generation tasks within a unified MLLM paradigm - from the diversity required in text-to-image generation to the precise controllability needed in image manipulation. In this work, we propose PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA unifies multi-granular visual features as both inputs and outputs of MLLMs, elegantly addressing the different granularity requirements of various image generation tasks within a unified MLLM framework. Following multimodal pretraining and task-specific instruction tuning, PUMA demonstrates proficiency in a wide range of multimodal tasks. This work represents a significant step towards a truly unified MLLM capable of adapting to the granularity demands of various visual tasks. The code and model will be released in https://github.com/rongyaofang/PUMA.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2024-10-17 |
| 分类 | cs.CV |
| 链接 | arXiv |