Chameleon
📌 核心贡献
Chameleon 是一个早期融合的、基于 token 的混合模态基础模型家族，能够任意顺序理解和生成图像与文本。其主要创新点在于提出了一种稳定的端到端训练方法、对齐策略以及专为这种早期融合混合模态设置定制的架构参数化。
📖 摘要
We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | Meta Fair |
| 发表 | 2024-05-16 |
| 分类 | cs.CL |
| 链接 | arXiv |