DINO
📌 核心贡献
提出 DINO 自监督方法（self-distillation with no labels），使 ViT 学到的特征天然包含语义分割信息，k-NN 分类达 78.3% top-1，远超 CNN。DINO 特征被后续大量视觉 tokenizer（如 iFSQ、ILLUME 等）用于提供判别性语义监督信号。
📖 摘要
In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | Facebook AI Research / Inria |
| 发表 | 2021-04-29 |
| 分类 | cs.CV |
| 链接 | arXiv |
📝 我的笔记
ICCV 2021. 前驱为 DINOv2 (2304.07193)，已在库。