---
license: apache-2.0
base_model: HuggingFaceBio/Carbon-500M
library_name: peft
tags:
- biology
- genomics
- dna
- promoter
- expression
- carbon
- lora
- peft
- adapter
- safetensors
- sequence-regression
---

# Carbon-500M LoRA for promoter expression ranking

This repository contains three PEFT LoRA adapters for `HuggingFaceBio/Carbon-500M`, trained to rank synthetic yeast promoter sequences by expression under the Random Promoter DREAM Challenge 2022 protocol.

The intended use is **promoter-expression ranking**. The scalar output is a train-normalized expression score and should not be treated as a well-calibrated absolute expression measurement.

Associated reproducibility package: https://github.com/muted-color/carbon-promoter-expression

## Contents

- `seed_20260523/adapter_model.safetensors`
- `seed_20260524/adapter_model.safetensors`
- `seed_20260525/adapter_model.safetensors`
- `metrics/final_test_metrics.csv`
- `metrics/freeze_manifest.json`
- `label_stats.json`
- `release_manifest.json`

The Carbon-500M base weights are not included. Load them from `HuggingFaceBio/Carbon-500M`.

## Results

Locked final-test set: 71,103 rows. The final-test read was performed once after writing `metrics/freeze_manifest.json`.

| Model | Spearman rho |
|---|---:|
| simple ridge | 0.585197 |
| k-mer 1-4 ridge | 0.710897 |
| Carbon LoRA seed 20260523 | 0.778652 |
| Carbon LoRA seed 20260524 | 0.790764 |
| Carbon LoRA seed 20260525 | 0.771563 |
| Carbon LoRA 3-seed mean prediction | 0.783603 |

The 3-seed mean prediction improves over the k-mer baseline by +0.072706 Spearman on the locked final test.

## Training setup

- Base model: `HuggingFaceBio/Carbon-500M`
- Task wrapper: `LlamaForSequenceClassification`
- LoRA rank: 8
- LoRA alpha: 16
- LoRA dropout: 0.05
- Target modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`
- Saved head: `score`
- Max length: 256
- Input format: `"<dna>" + sequence`
- Tokenizer setting: `add_special_tokens=False`
- Train rows: 1,000,000
- Epochs: 1
- Primary metric: Spearman rank correlation

## Loading

This is an adapter repository, not a standalone Transformers pipeline model. Load the Carbon-500M base model first, then load one seed adapter.

```python
import torch
from peft import PeftModel
from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

base_id = "HuggingFaceBio/Carbon-500M"
adapter_id = "soleaf/carbon-500m-lora-promoter-expression-rank"
seed = "seed_20260524"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token or tokenizer.unk_token

config = AutoConfig.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=1,
    problem_type="regression",
)
config.pad_token_id = tokenizer.pad_token_id

model = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    config=config,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_id, subfolder=seed)
model.eval()
```

`score.weight` is newly initialized when the base model is loaded as a sequence-classification model; this is expected. The trained regression head is loaded from the adapter in the next step.

## Scoring example

```python
sequences = [
    "ACGTACGTACGT",
    "TATAAAACGTACGT",
]

inputs = tokenizer(
    ["<dna>" + seq for seq in sequences],
    padding=True,
    truncation=True,
    max_length=256,
    return_tensors="pt",
    add_special_tokens=False,
)

with torch.no_grad():
    pred_z = model(**inputs).logits.squeeze(-1).float()

# Higher values should be interpreted as higher predicted expression rank.
ranking = torch.argsort(pred_z, descending=True)
```

To map the output back onto the training expression scale, use `label_stats.json`:

```python
expression_mean = 11.145822088831892
expression_std = 2.371304223468006
pred_expression = pred_z * expression_std + expression_mean
```

The expression-scale conversion is provided for convenience; rank-based use is the evaluated setting.

## Scope and limitations

- The adapter was evaluated for this synthetic yeast promoter-expression protocol, not as a general cross-species promoter model.
- Use the score primarily for ranking. Absolute expression calibration was poor on the shifted final-test distribution.
- Use the same input/windowing protocol as training. A phase/window diagnostic found meaningful sensitivity to off-protocol perturbations.
- The provided adapters are the 1M-label runs. Separate low-label LoRA experiments at 6,044 and 60,436 rows underperformed matched k-mer controls.

## Data and provenance

Raw and processed datasets are not redistributed in this model repository.

- Dataset: Random Promoter DREAM Challenge 2022, Zenodo DOI `10.5281/zenodo.10633252`
- Dataset license: Creative Commons Attribution 4.0 International
- Base model: `HuggingFaceBio/Carbon-500M`, Apache-2.0
- Reproducibility package: https://github.com/muted-color/carbon-promoter-expression