soleaf/carbon-500m-lora-promoter-expression-rank

Carbon-500M LoRA for promoter expression ranking

This repository contains three PEFT LoRA adapters for HuggingFaceBio/Carbon-500M, trained to rank synthetic yeast promoter sequences by expression under the Random Promoter DREAM Challenge 2022 protocol.

The intended use is promoter-expression ranking. The scalar output is a train-normalized expression score and should not be treated as a well-calibrated absolute expression measurement.

Associated reproducibility package: https://github.com/muted-color/carbon-promoter-expression

Contents

• seed_20260523/adapter_model.safetensors
• seed_20260524/adapter_model.safetensors
• seed_20260525/adapter_model.safetensors
• metrics/final_test_metrics.csv
• metrics/freeze_manifest.json
• label_stats.json
• release_manifest.json

The Carbon-500M base weights are not included. Load them from HuggingFaceBio/Carbon-500M.

Results

Locked final-test set: 71,103 rows. The final-test read was performed once after writing metrics/freeze_manifest.json.

Model	Spearman rho
simple ridge	0.585197
k-mer 1-4 ridge	0.710897
Carbon LoRA seed 20260523	0.778652
Carbon LoRA seed 20260524	0.790764
Carbon LoRA seed 20260525	0.771563
Carbon LoRA 3-seed mean prediction	0.783603

The 3-seed mean prediction improves over the k-mer baseline by +0.072706 Spearman on the locked final test.

Training setup

• Base model: HuggingFaceBio/Carbon-500M
• Task wrapper: LlamaForSequenceClassification
• LoRA rank: 8
• LoRA alpha: 16
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj
• Saved head: score
• Max length: 256
• Input format: "<dna>" + sequence
• Tokenizer setting: add_special_tokens=False
• Train rows: 1,000,000
• Epochs: 1
• Primary metric: Spearman rank correlation

Loading

This is an adapter repository, not a standalone Transformers pipeline model. Load the Carbon-500M base model first, then load one seed adapter.

import torch
from peft import PeftModel
from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

base_id = "HuggingFaceBio/Carbon-500M"
adapter_id = "soleaf/carbon-500m-lora-promoter-expression-rank"
seed = "seed_20260524"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token or tokenizer.unk_token

config = AutoConfig.from_pretrained(
    base_id,
    trust_remote_code=True,
    num_labels=1,
    problem_type="regression",
)
config.pad_token_id = tokenizer.pad_token_id

model = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    config=config,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_id, subfolder=seed)
model.eval()

score.weight is newly initialized when the base model is loaded as a sequence-classification model; this is expected. The trained regression head is loaded from the adapter in the next step.

Scoring example

sequences = [
    "ACGTACGTACGT",
    "TATAAAACGTACGT",
]

inputs = tokenizer(
    ["<dna>" + seq for seq in sequences],
    padding=True,
    truncation=True,
    max_length=256,
    return_tensors="pt",
    add_special_tokens=False,
)

with torch.no_grad():
    pred_z = model(**inputs).logits.squeeze(-1).float()

# Higher values should be interpreted as higher predicted expression rank.
ranking = torch.argsort(pred_z, descending=True)

To map the output back onto the training expression scale, use label_stats.json:

expression_mean = 11.145822088831892
expression_std = 2.371304223468006
pred_expression = pred_z * expression_std + expression_mean

The expression-scale conversion is provided for convenience; rank-based use is the evaluated setting.

Scope and limitations

• The adapter was evaluated for this synthetic yeast promoter-expression protocol, not as a general cross-species promoter model.
• Use the score primarily for ranking. Absolute expression calibration was poor on the shifted final-test distribution.
• Use the same input/windowing protocol as training. A phase/window diagnostic found meaningful sensitivity to off-protocol perturbations.
• The provided adapters are the 1M-label runs. Separate low-label LoRA experiments at 6,044 and 60,436 rows underperformed matched k-mer controls.

Data and provenance

Raw and processed datasets are not redistributed in this model repository.

• Dataset: Random Promoter DREAM Challenge 2022, Zenodo DOI 10.5281/zenodo.10633252
• Dataset license: Creative Commons Attribution 4.0 International
• Base model: HuggingFaceBio/Carbon-500M, Apache-2.0
• Reproducibility package: https://github.com/muted-color/carbon-promoter-expression