[-4c72b0?style=flat-square)](#results) [](#results) [](#quickstart) [](LICENSE) [How it works](#how-it-works) · [Quickstart](#quickstart) · [Results](#results) · [Configuration](#configuration) · [Notes from development](#notes-from-development)
The semantic channel fans out across snippet search, keyword paper search, hypothetical-abstract (HyDE) probes, and citation expansion, then fuses everything with reciprocal rank fusion. The query is decomposed into explicit relevance criteria, and a gpt-4o-mini pool judge scores the top 400 candidates against each criterion separately, with position-decay weighting (later auto-derived criteria are noisier, so they count less). Pointwise scores strand real matches in the ambiguous middle, so a gpt-5-mini listwise tournament reorders that contested band with sliding windows, pooled Borda aggregation across passes, and a bounded demotion cap so one bad window cannot destroy a good paper's rank.
### Metadata search
"Papers at CHI after 2020 citing NeurIPS papers" is a database query, not a similarity search. The metadata channel compiles the query into a typed plan (venue set, year filter, citation set) and executes the intersection deterministically, with venue canonicalization and acronym expansion on both sides. When the local citation graph is present, venue-to-venue citation constraints run against it instead of rate-limited APIs; other citation constraints use the corpus API.
### Specific lookup
When the query names one paper ("the AlphaFold 2 Nature paper"), IRIS extracts every clue it can (title fragments, authors, year, topic, artifacts), scores candidates against all of them, and walks references when direct resolution fails. It returns its best match: one paper in the typical case, a small hedged set when the name is genuinely ambiguous, with verbatim evidence either way.
## Quickstart
```bash
git clone https://github.com/rasynai/rasyn-iris.git
cd rasyn-iris
pip install astabench==0.5.4
```
Create `iris_asta/.env` (see [`.env.example`](iris_asta/.env.example)):
```bash
OPENAI_API_KEY=sk-...
ASTA_TOOL_KEY=... # free at https://api.semanticscholar.org
```
Run the benchmark under the official harness with the leaderboard configuration:
```bash
PFBMAX_CJ_POOL=1 PFBMAX_CJ_POOL_DEPTH=400 PFBMAX_CJ_MODEL=gpt-4o-mini \
PFBMAX_CJ_POSDECAY=0.6 PFBMAX_TOURN=1 PFBMAX_TOURN_MODEL=gpt-5-mini \
inspect eval astabench/paper_finder_validation \
--solver pfbmax/inspect_entry.py@pfbmax_solver \
--model openai/gpt-4o-mini
```
Our own live-retrieval runs of this command scored 0.357 and 0.366. Read
"Performance and rate limits" below before you start: on the public corpus API
this is not a quick command.
### Performance and rate limits
Everything here is measured, on our hardware, in August 2026:
| What | Observed |
|---|---|
| One metadata query (venue, year, citation constraints) | about 6 seconds |
| One specific-paper lookup | about 1 minute |
| One semantic query, cold cache, public API | did not finish within 15 minutes |
| A single corpus call | 1 to 30 seconds, highly variable |
The semantic channel deliberately issues dozens of corpus calls per query
(rephrasings, HyDE probes, citation expansion) because pool recall is the
ceiling on the score. On the free public endpoint that is slow, and it can
stall entirely when you are being rate limited. Nothing is hung; it is
waiting on the API. Practical advice:
- The read-through cache in `pfbmax/cache/` makes repeat runs much faster, and
the certified runs above were served from a warmed cache.
- If you have Asta MCP access, `PFBMAX_USE_MCP=1` routes corpus calls through
the gateway instead of the public endpoint.
- For a quick smoke test rather than a scored run, shrink the fanout:
`PFBMAX_MAX_REPHRASINGS=2 PFBMAX_MAX_CRITERIA=3 PFBMAX_MAX_HYDE=1
PFBMAX_MAX_CALLS=12`. Expect a much weaker result; this is for checking that
your keys and wiring work.
- Missing or invalid keys now fail immediately with a message rather than
producing a silent zero-score run.
Or call IRIS from your own code:
```python
import os, sys
sys.path += ["pfbmax", "iris_asta"]
for line in open("iris_asta/.env"):
if "=" in line and not line.startswith("#"):
k, v = line.strip().split("=", 1)
os.environ.setdefault(k, v)
from iris_asta.asta_client import AstaClient
from iris_asta.config import load_config
from llm import LLM
import router
client = AstaClient(load_config())
results = router.solve(
"diffusion models for protein structure generation evaluated on CASP, after 2022",
client, LLM(), inserted_before=None,
)
for paper_id, evidence in results:
print(paper_id, evidence[:100])
```
`router.solve` never raises. It routes, retrieves, judges, and returns `[(corpus_id, verbatim_evidence), ...]` best first. A semantic query fans out into many corpus calls, so with a fresh key expect a first run to take several minutes; if the public API is rate-limiting you hard, set `PFBMAX_USE_MCP=1`.
## Configuration
Everything is tunable by environment variable. The configuration of the certified runs:
| Variable | Value | What it does |
|---|---|---|
| `PFBMAX_CJ_POOL` | `1` | enable the per-criterion pool judge |
| `PFBMAX_CJ_MODEL` | `gpt-4o-mini` | pool judge model (unset, the code default is gpt-4o-mini) |
| `PFBMAX_CJ_POOL_DEPTH` | `400` | candidates judged per query |
| `PFBMAX_CJ_POSDECAY` | `0.6` | criterion position-decay weight |
| `PFBMAX_TOURN` | `1` | enable the listwise tournament (unset any of these to disable) |
| `PFBMAX_TOURN_MODEL` | `gpt-5-mini` | tournament ranking model |
| `PFBMAX_TOURN_PASSES` | `2` | tournament passes over the contested band |
| `PFBMAX_TOURN_PRIOR` | `0.4` | blend weight of the pointwise prior |
| `PFBMAX_TOURN_DEMOTE_CAP` | `8` | max ranks a paper can fall per tournament |
| `PFBMAX_TOPIC_MODEL` | `gpt-4o-2024-11-20` | topic-verification model on the specific-paper channel (default) |
| `PFBMAX_USE_MCP` | unset | set to `1` to route corpus calls via Asta's MCP gateway (rate-limit fallback) |
| `PFBMAX_CITEGRAPH` / `PFBMAX_PMETA` | *paths* | optional local citation graph + metadata SQLite (build tooling not included) |
The certified runs also widened the retrieval fanout beyond the shipped
defaults. For completeness, that environment was:
| Variable | Value | What it does |
|---|---|---|
| `PFBMAX_CJ_WORKERS` | `8` | parallel judge requests |
| `PFBMAX_MAX_CALLS` | `220` | corpus-call budget per query |
| `PFBMAX_SEEDS` / `PFBMAX_SEED_POOL` | `10` / `20` | citation-expansion seeds and the window they come from |
| `PFBMAX_LIMIT_SNIPPET_RAW` | `150` | snippet depth for the raw query |
| `PFBMAX_LIMIT_SNIPPET_HYDE` | `150` | snippet depth per HyDE probe |
| `PFBMAX_LIMIT_CITATIONS` | `120` | citation rows per direction per seed |
| `PFBMAX_FANOUT_WORKERS` | `5` | parallel corpus probes |
| `PFBMAX_FETCH_RETRIES` | `2` | retries per corpus call |
| `PFBMAX_CACHE_ONLY` | `1` | serve corpus reads only from the local cache |
## Repository layout
| Path | What lives there |
|---|---|
| [`pfbmax/router.py`](pfbmax/router.py) | query classification and channel dispatch, start here |
| [`pfbmax/criterion_judge.py`](pfbmax/criterion_judge.py) | the per-criterion pool judge |
| [`pfbmax/tournament.py`](pfbmax/tournament.py) | listwise tournament reranker |
| [`pfbmax/metadata_solver.py`](pfbmax/metadata_solver.py) | deterministic metadata plans and citation graph execution |
| [`pfbmax/inspect_entry.py`](pfbmax/inspect_entry.py) | official harness entry point |
| [`iris_asta/`](iris_asta/) | corpus client, config, rate limiting, snapshot date enforcement |
## Notes from development
Every component here earned its place through a controlled experiment; losers were deleted or left flag-gated off. The campaign kept a written ledger (23 entries) and finished with six official harness runs. Things that did not work, so you do not have to retry them:
- Trained cross-encoder rerankers (several variants): never beat the LLM judge
- Bradley-Terry and PageRank aggregation: pooled Borda won
- Hierarchical tournaments: flat sliding windows won
- Permissive judge prompts, evidence enrichment, prior blends at admission: all net negative
Things that paid: position-decay criterion weighting, the tournament package (pooled cross-pass Borda + pointwise prior + demotion cap), and the local citation graph.
The demotion cap is a good example of the approach. Pilot runs showed one rescue worth +0.094 but two catastrophic demotions, so ascent is unlimited and descent is capped at 8 ranks. Pooled Borda plus that asymmetry is what took the tournament from net zero to net positive.
## Contributing
Issues and PRs welcome. One rule: no change lands without a measured comparison under the official harness (`inspect eval astabench/paper_finder_validation ...`). Post the before/after in the PR.
## License
[MIT](LICENSE) © 2026 [Rasyn AI](https://rasyn.ai). AstaBench (benchmark, harness, grading) is Ai2's work, used as a pip dependency. One exception to "no Ai2 content in this repo": the pool judge intentionally reproduces the harness's Apache-2.0 judging prompt so that selection matches grading; see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for the attribution and license text.
---