README.md

---
base_model: Qwen/Qwen3-8B
library_name: peft
license: apache-2.0
tags:
  - reinforcement-learning
  - lora
  - systems
  - kernel-tuning
---

# reef-nccl-ep-qwen3-8b

A Qwen3-8B LoRA adapter (rank 32) trained with reinforcement learning to propose
high-bandwidth configurations for an expert-parallel collective-communication kernel
(MoE dispatch/combine over RDMA). This is the final checkpoint of a harness/model
co-evolution experiment built on [reef](https://github.com/Human-Agent-Society/reef)
with the slime/Megatron training backend and SGLang serving.

## What it does

Given a task description of the kernel's five tuning parameters (RDMA queue pairs,
tokens per chunk, dispatch sub-puts, combine streaming batch, communication SMs), the
model proposes a configuration as a JSON object. Every reward during training was a
hardware measurement: dispatch+combine bandwidth in GB/s on a 16-GPU H200 system, gated
by a 16/16 correctness test. No synthetic rewards.

## Results

- The model's sampling attractor is the measured optimum at 8192 tokens
  (qps=2, chunk=256, subputs=6, cb=32, sms=11 -> 192.99 GB/s), which matches the best
  human-guided result on the same judge.
- During training the system found configs no one had measured before, including
  qps=2, chunk=256, subputs=16, cb=64, sms=12 -> 185.76 GB/s and, at 16384 tokens
  (a size never previously tuned), qps=2, chunk=512, subputs=8, cb=32, sms=11 ->
  191.45 GB/s.
- The trained 8B outperforms a 480B coding model prompted zero-shot on the same task
  and judge (480B best: 147.46 GB/s).

## How it was trained

Alternating coordinate ascent between the model and its harness, with the judge frozen:

1. Model phase: 16 steps of grouped relative-reward RL (reef TTTD recipe, slime/Megatron
   LoRA backend, 4 groups x 4-8 rollouts per step; rewards from the hardware judge).
2. Harness phase: a 480B model proposes revisions to the task instruction from training
   evidence (reward curves, measured episodes); candidates are scored on a frozen
   checkpoint under an equal sample budget, and a revision is adopted only on a clear
   win. The instruction carries measured heuristics forward; the judge never changes.

Five model phases and three harness phases produced this checkpoint. Intermediate
checkpoints of every run are published in the companion repos
(reef-nccl-ep-qwen3-8b-lora-coevolve-run2/-run4/-run5, -16k-run2, -discovery-16step).

## Loading

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base, "jhhj25/reef-nccl-ep-qwen3-8b")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
```

Disable thinking mode when sampling configs
(`chat_template_kwargs={"enable_thinking": False}`), and prompt with a task description
of the parameter space; the model answers with a single JSON config block.