Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
📌 核心贡献
Janus引入了一个自回归框架，通过解耦视觉编码为独立的路径，同时仍利用统一的Transformer架构进行处理，从而统一了多模态理解和生成。该方法旨在解决现有方法（如Chameleon）在视觉编码上对理解和生成任务使用单一视觉编码器导致的次优性能，尤其是在多模态理解方面。
📖 摘要
In this paper, we introduce Janus, an autoregressive framework that unifies multimodal understanding and generation. Prior research often relies on a single visual encoder for both tasks, such as Chameleon. However, due to the differing levels of information granularity required by multimodal understanding and generation, this approach can lead to suboptimal performance, particularly in multimodal understanding. To address this issue, we decouple visual encoding into separate pathways, while still leveraging a single, unified transformer architecture for processing. The decoupling not only alleviates the conflict between the visual encoder’s roles in understanding and generation, but also enhances the framework’s flexibility. For instance, both the multimodal understanding and generation components can independently select their most suitable encoding methods. Experiments show that Janus surpasses previous unified model and matches or exceeds the performance of task-specific models. The simplicity, high flexibility, and effectiveness of Janus make it a strong candidate for next-generation unified multimodal models.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2024-10-17 |
| 分类 | cs.CV, cs.AI |
| 链接 | arXiv |