NLIP: NLIP: Noise-robust Language-Image Pre-training
📌 核心贡献
针对大规模跨模态预训练数据中普遍存在的错误和不完整噪音问题，现有方法未能同时有效解决。本文提出NLIP框架，通过自动挖掘现有数据，无需人工规则或生成伪目标，即可鲁棒地稳定预训练过程。
📖 摘要
Large-scale cross-modal pre-training paradigms have recently shown ubiquitous success on a wide range of downstream tasks, e.g., zero-shot classification, retrieval and image captioning. However, their successes highly rely on the scale and quality of web-crawled data that naturally contain incomplete and noisy information (e.g., wrong or irrelevant content). Existing works either design manual rules to clean data or generate pseudo-targets as auxiliary signals for reducing noise impact, which do not explicitly tackle both the incorrect and incomplete challenges simultaneously. In this paper, to automatically mitigate the impact of noise by solely mining over existing data, we propose a principled Noise-robust Language-Image Pre-training framework (NLIP) to stabilize pre-training via two schemes: noise-harmonization and noise-completion.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 发表 | 2022-12-14 |
| 分类 | cs.CV |
| 链接 | arXiv |