GRAM-R2
📌 核心贡献
提出自训练方法 GRAM-R2，利用无标注数据激发生成式奖励模型的奖励推理能力，产出偏好标签和推理理由，可作为奖励推理的基础模型在多种任务上零/少样本适配。
📖 摘要
Significant progress in reward modeling over recent years has been driven by a paradigm shift from task-specific designs towards generalist reward models. Despite this trend, developing effective reward models remains a fundamental challenge: the heavy reliance on large-scale labeled preference data. Pre-training on abundant unlabeled data offers a promising direction, but existing approaches fall short of instilling explicit reasoning into reward models. To bridge this gap, we propose a self-training approach that leverages unlabeled data to elicit reward reasoning in reward models. Based on this approach, we develop GRAM-R2, a generative reward model trained to produce not only preference labels but also accompanying reward rationales. GRAM-R2 can serve as a foundation model for reward reasoning and can be applied to a wide range of tasks with minimal or no additional fine-tuning. It can support downstream applications such as response ranking and task-specific reward tuning. Experiments on response ranking, task adaptation, and reinforcement learning from human feedback demonstrate that GRAM-R2 consistently delivers strong performance, outperforming several strong discriminative and generative baselines.
🔍 关键信息
| 字段 | 内容 |
|---|---|
| 机构 | - |
| 发表 | 2025-09-02 (AAAI 2026) |
| 分类 | cs.CL, cs.LG |
| 链接 | arXiv |
📝 我的笔记
动机与问题定义
现有奖励模型的核心挑战：
- 数据依赖：高度依赖大规模标注偏好数据
- 缺乏推理：现有预训练方法无法为奖励模型注入显式推理能力
- 泛化性：任务特定的 RM 难以迁移到新任务
GRAM-R2 的目标是构建一个可作为基础模型的生成式奖励模型，通过自训练在无标注数据上学习奖励推理。
方法细节
核心思路： 自训练 (Self-Training)
- 利用无标注数据激发模型的奖励推理能力
- 模型不仅输出偏好标签，还生成推理理由 (reward rationales)
关键特点：
- 生成式奖励模型：同时输出偏好判断和推理过程
- 基础模型特性：可在多种下游任务上进行零/少样本适配
- 支持响应排序、任务特定奖励微调、RLHF 等应用
关键实验结果
GRAM-R2 在三个维度上展示了强劲性能：
- 响应排序：超越多个强判别式和生成式基线
- 任务适配：在少量或零额外微调下适配新任务
- RLHF 应用：作为 RLHF 奖励信号时表现一致且强劲
总结
GRAM-R2 提出了一个有前景的方向：通过自训练在无标注数据上学习奖励推理，构建生成式奖励基础模型。被 AAAI 2026 接收，说明其方法论和实验结果获得了认可。这一方向与 GenRM 和 DeepSeek-GRM 的生成式奖励建模思路一脉相承，但更强调无标注数据的利用和推理能力的培养。
🔗 相关论文
基于/改进自： GenRM
同方向： DeepSeek-GRM