Beyond Language Modeling: An Exploration of Multimodal Pretraining
📌 核心贡献
从零开始的受控多模态预训练实验，基于 Transfusion 框架系统探索设计空间。四大发现：RAE 是最优视觉表征、视觉与语言数据互补、统一预训练自然涌现世界模型能力、MoE 高效调和视觉数据饥渴性与语言模型容量需求。
📖 摘要
We provide empirical clarity through controlled, from-scratch pretraining experiments on multimodal models, using the Transfusion framework. Key findings: (i) RAE provides optimal unified visual representation; (ii) visual and language data are complementary; (iii) unified multimodal pretraining leads naturally to world modeling; (iv) MoE enables efficient multimodal scaling. Vision is significantly more data-hungry than language.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2026-03-03 |
| 机构 | Meta AI · NYU · FAIR |
| 分类 | cs.CV |
| 链接 | arXiv |