Generative Multimodal Models are In-Context Learners
📌 核心贡献
Emu2是一个拥有370亿参数的生成式多模态模型，通过大规模多模态序列的统一自回归目标训练，显著增强了多模态模型在上下文学习方面的能力。该模型展现出强大的多模态上下文学习能力，即便在只提供少量示例或简单指令的情况下也能有效解决多模态任务。
📖 摘要
The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2023-12-20 |
| 分类 | cs.CV |
| 链接 | arXiv |