DreamLLM: Synergistic Multimodal Comprehension and Creation
📌 核心贡献
DreamLLM提出了一种学习框架，通过语言和图像后验的生成建模，直接在原始多模态空间中采样，从而实现多模态大语言模型 (MLLM) 在多模态理解和创作之间的协同作用。它旨在克服外部特征提取器的局限性，从而获得更彻底的多模态理解。
📖 摘要
This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM’s superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2023-09-20 |
| 分类 | cs.CV, cs.CL |
| 链接 | arXiv |