ILLUME
📌 核心贡献
提出统一多模态大语言模型 ILLUME，通过包含语义信息的视觉 tokenizer 和渐进式多阶段训练，将预训练数据量缩减至 15M（比常规少 4 倍以上），同时性能与 Janus 等 SOTA 方法持平或更优。提出自增强多模态对齐方案，让模型自评一致性以协同提升理解和生成能力。
📖 摘要
In this paper, we introduce ILLUME, a unified multimodal large language model (MLLM) that seamlessly integrates multimodal understanding and generation capabilities within a single large language model through a unified next-token prediction formulation. To address the large dataset size typically required for image-text alignment, we propose to enhance data efficiency through the design of a vision tokenizer that incorporates semantic information and a progressive multi-stage training procedure. This approach reduces the dataset size to just 15M for pretraining — over four times fewer than what is typically needed — while achieving competitive or even superior performance with existing unified MLLMs, such as Janus. Additionally, to promote synergistic enhancement between understanding and generation capabilities, which is under-explored in previous works, we introduce a novel self-enhancing multimodal alignment scheme. This scheme supervises the MLLM to self-assess the consistency between text descriptions and self-generated images, facilitating the model to interpret images more accurately and avoid unrealistic and incorrect predictions caused by misalignment in image generation. Based on extensive experiments, our proposed ILLUME stands out and competes with state-of-the-art unified MLLMs and specialized models across various benchmarks for multimodal understanding, generation, and editing.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | Huawei Noah’s Ark Lab |
| 发表 | 2024-12-09 |
| 分类 | cs.CV |
| 链接 | arXiv |
📝 我的笔记
🔗 相关论文
基于/改进自： VQGAN 同方向： Chameleon, SEED-LLaMA