NEPA
⚠️ arxiv_id 待确认，论文名可能有误，请手动核实
📌 核心贡献
这篇论文受自然语言生成式预训练的启发，提出了一种新的自监督视觉学习方法。其核心创新在于，模型不再仅仅学习表征，而是直接学习预测嵌入（embeddings）。具体来说，模型通过因果掩码（causal masking）和停止梯度（stop gradient）机制，预测未来图像块的嵌入，从而直接完成预测任务。
📖 摘要
Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | |
| 发表 | 2025-12-18 |
| 分类 | cs.CV |
| 链接 | arXiv |