Liquid: Language Models are Scalable and Unified Multi-modal Generators
📌 核心贡献
Liquid提出了一个自回归生成范式，通过将图像分词为离散代码并在共享特征空间中与文本令牌一起学习这些代码嵌入，从而无缝整合视觉理解和生成。该模型首次揭示了统一训练不可避免地带来的性能下降规律，旨在实现可扩展且统一的多模态生成。
📖 摘要
We present Liquid, an auto-regressive generation paradigm that seamlessly integrates visual comprehension and generation by tokenizing images into discrete codes and learning these code embeddings alongside text tokens within a shared feature space for both vision and language. Unlike previous multimodal large language model (MLLM), Liquid achieves this integration using a single large language model (LLM), eliminating the need for external pretrained visual embeddings such as CLIP. For the first time, Liquid uncovers a scaling law that performance drop unavoidably brought by the unified training of visual and language tasks diminishes as the model size increases. Furthermore, the unified token space enables visual generation and comprehension tasks to mutually enhance each other, effectively removing the typical interference seen in earlier models. We show that existing LLMs can serve as strong foundations for Liquid, saving 100x in training costs while outperforming Chameleon in multimodal capabilities and maintaining language performance comparable to mainstream LLMs like LLAMA2. Liquid also outperforms models like SD v2.1 and SD-XL (FID of 5.47 on MJHQ-30K), excelling in both vision-language and text-only tasks. This work demonstrates that LLMs such as Qwen2.5 and GEMMA2 are powerful multimodal generators, offering a scalable solution for enhancing both vision-language understanding and generation. The code and models will be released at https://github.com/FoundationVision/Liquid.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2024-12-05 |
| 分类 | cs.CV |
| 链接 | arXiv |