Emu: Generative Pretraining in Multimodality
📌 核心贡献
Emu是一个基于Transformer的多模态基础模型，能够无缝生成多模态上下文中的图像和文本。它通过一个统一的自回归训练过程，将视觉信号编码为嵌入，并与文本令牌共同构成交错的输入序列，旨在实现对任何单模态或多模态数据输入的无差别处理。
📖 摘要
We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu can serve as a generalist multimodal interface for both image-to-text and text-to-image tasks, and supports in-context image and text generation. Across a broad range of zero-shot/few-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to state-of-the-art large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2023-07-11 |
| 分类 | cs.CV |
| 链接 | arXiv |