ESM Atlas (ESMFold2) distilled structures for MarinFold training

A quality-filtered, novelty-deduplicated subset of the ESM Atlas (the "ESMFold2 Atlas", ~1.1B predicted monomer structures from Biohub's Language Modeling Materializes a World Model of Protein Biology, bioRxiv 2026.06.03.729735), materialized to mmCIF for MarinFold training-set expansion.

Each row is one cluster representative: its decoded 3D structure (mmCIF text, per-residue pLDDT in the B-factor column), its sequence, and the selection metadata that put it here. Built for MarinFold issue #91. The column layout is aligned to timodonnell/afdb-24M so the two datasets union cleanly into one training corpus.

• Representatives: 66,759,963 (one per structural cluster, MMseqs2 linclust @ 40% id)
• Size: ~1.9 TiB across 3,338 Parquet parts of 20,000 structures each (the last part holds 19,963)
• Content: ESMFold2 monomer structures only (one chain per row)

Contents

structures/parts/part_NNNNN.parquet   # 3,338 parts, 20k structures each; the payload
selected_manifest.csv                  # 66,759,963 rows: the authoritative rep list + scores
README.md                              # this file

selected_manifest.csv is the funnel's deliverable (columns cluster_id, protein_hash, seq_len, mean_plddt, ptm, plddt_std, cluster_size); it joins to the parts on protein_hash == entry_id if you want to reconcile the selection with the materialized structures.

Schema (per part)

Names align with timodonnell/afdb-24M where the field exists (so a union is a straight concat); the rest are Atlas-specific extras.

Caveats for anyone unioning this with afdb-24M or training on it:

• pLDDT scale. global_plddt here is 0-1 (Atlas convention). afdb-24M's same-named column may be 0-100. Rescale one side before training on the union.
• split is a placeholder. Everything is "train". A real train/val/test holdout is deferred to a later decision coordinated with MarinFold's (unbuilt) eval set and its leakage dedup — do not treat this as a validated split.
• afdb-only columns are absent, not fabricated. uniprot_accession, tax_id, organism_name, struct_cluster_id, gcs_uri do not exist for Atlas rows and are omitted rather than filled with nulls/guesses.
• seq_ok is essentially all True (~1 mismatch per 20k spot-checked — a trace integrity signal, not a filter that was applied).

Provenance

Derived from s3://esm-protein-atlas/v1/folds/folds_1B.lance (1,095,530,880 rows; structures predicted with esmfold2-exp-2026-03, 3 recycles / 22 diffusion steps). The 1.1B Atlas monomers were reduced to these 66.76M representatives by a five-stage funnel:

Then each selected rep's structure_blob was decoded (brotli + msgpack atom37 → mmCIF) and written to the parts here.

• Source dataset: ESM Atlas v1 folds_1B.lance (AWS Open Data, us-west-2)
• Novelty/leakage reference: afdb-24M cluster reps via the exp41 foldseek DB (silterra/afdb-24M-foldseek-train-reps)
• Builder: experiments/exp91_evals_esmfold2_atlas/ in the MarinFold repo (pipeline.py funnel → materialize.py decode → upload_to_hf.py publish), run on AWS us-west-2, 2026-07.

Use

import pyarrow.dataset as ds

# Stream the parts directly from the bucket (or `hf buckets sync` them local first).
data = ds.dataset("hf://buckets/<owner>/<bucket>/structures/parts", format="parquet")
batch = data.head(4)
cif = batch.column("cif_content")[0].as_py()   # parse with gemmi / biotite

This is the input format marinfold contacts-v1 generate reads (entry_id + cif_content), so it feeds document generation unchanged.

License

The ESM Atlas is distributed under CC BY-SA 4.0 per the AWS Open Data registry (the paper PDF states CC BY — the two disagree). This derived dataset is redistributed under the same CC BY-SA 4.0 with attribution to the ESM Atlas authors (Biohub). ShareAlike applies: downstream redistribution must carry the same license.

Before any public/first-class release, verify the exact data-license terms on the canonical source (registry vs. paper) and set the repo license accordingly.