---
base_model: deepseek-ai/DeepSeek-R1-0528-Qwen3-8B
datasets:
- DORAEMONG/PRO-STEP-PRM-Data
language:
- en
library_name: peft
license: mit
pipeline_tag: text-generation
tags:
- process-reward-model
- prm
- retrieval-augmented-generation
- lora
---

# PRO-STEP: Process Reward Model for Agentic RAG

**Paper**: [PRO-STEP: Step-level Process Reward Optimization for Retrieval-Augmented Generation](https://huggingface.co/papers/2609.01658)  
**Code**: [github.com/keemminnke/PRO-Step](https://github.com/keemminnke/PRO-Step)

The supervised step-level **Process Reward Model (PRM)** for [PRO-STEP](https://huggingface.co/DORAEMONG/PRO-STEP-Policy-7B). Trained to evaluate agentic-RAG steps along six axes (entity grounding, search query quality, reasoning, answer specificity, recovery, overconfidence).

- **Base model**: DeepSeek-R1-0528-Qwen3-8B
- **Adapter**: LoRA (r=16)
- **Training data**: [DORAEMONG/PRO-STEP-PRM-Data](https://huggingface.co/datasets/DORAEMONG/PRO-STEP-PRM-Data) — ~109K step annotations across 31,728 trajectories from 2,000 questions
- **Annotation source**: QwQ-32B (open-source labeler), 84% human agreement on a 50-trajectory inspection sample
- **Output**: rationale + binary GOOD/BAD label

## Performance vs other PRMs (BoN/WMV at K=128, F1)

| Setting | **PRO-STEP PRM** | VersaPRM | Math-PRM | Majority Voting |
|---|---|---|---|---|
| HotpotQA WMV-min | **59.50** ★ | 53.58 | 53.90 | 54.07 |
| HotpotQA BoN-min | **59.07** ★ | 53.26 | 48.07 | 54.07 |
| PopQA WMV-min | **50.01** ★ | 49.50 | 49.49 | 48.74 |
| PopQA BoN-min | **49.07** ★ | 45.11 | 41.32 | 48.74 |
| 2Wiki WMV-min | **46.01** ★ | 43.40 | 43.49 | 44.00 |
| 2Wiki BoN-min | 43.54 | 27.78 | 34.27 | **44.00** |

PRO-STEP PRM is the only learned PRM that stays at or above the Majority Voting baseline across 5 of 6 settings. VersaPRM and Math-PRM collapse below MV on multi-hop BoN.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-0528-Qwen3-8B", torch_dtype="auto")
prm = PeftModel.from_pretrained(base_model, "DORAEMONG/PRO-STEP-PRM-8B")
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-0528-Qwen3-8B")

# Inference: provide trajectory step, get [REASONING] + Label: 0/1
# See paper Appendix A for the MCTS scoring prompt
```

## Citation

```bibtex
@article{prostep2026,
  title={PRO-STEP: Step-level Process Reward Optimization for Retrieval-Augmented Generation},
  author={...},
  year={2026}
}
```