CLIP-ViT-L-14.laion2b-pt.planktonzilla-pt

CLIP-style (image–text) plankton model from the paper Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems (Contreras, Valenzuela, Martí & Sanchez-Pi, 2026). It is one arm of the paper's controlled comparison between supervised and CLIP-style training on a shared ViT backbone, trained with taxonomic lineage used as the text description (following the BioCLIP recipe).

• Architecture: CLIP ViT-L/14 (embed dim 768, 224×224 input)
• Initialised from: LAION-2B ViT-L/14 (LAION-2B)
• Trained on: the 3.74M-image plankton subset of Planktonzilla-17M (60/20/20 train/val/test, stratified by source dataset and taxonomic label)
• Recipe: OpenCLIP, 100 epochs, global batch 16,384, 64× H100; taxonomic-lineage captions
• Preprocessing: CLIP mean/std, bicubic, shortest-side resize

Results

Macro-F1 per taxonomic rank on the in-domain test split (Planktonzilla-17M plankton subset), from Table 2 of the paper. Models are selected on the highest validation Macro-F1.

Kingdom	Phylum	Class	Order	Family	Genus	Species
0.967	0.927	0.878	0.873	0.845	0.839	0.825

The paper reports that models trained on Planktonzilla-17M substantially outperform off-the-shelf BioCLIP / BioCLIP 2 across all taxonomic ranks, and that fully supervised classifiers on the same backbone match or slightly exceed these CLIP-style models.

Usage (open_clip)

import open_clip

hub = "hf-hub:project-oceania/CLIP-ViT-L-14.laion2b-pt.planktonzilla-pt"
model, _, preprocess = open_clip.create_model_and_transforms(hub)
tokenizer = open_clip.get_tokenizer(hub)

Use it for zero-/few-shot classification by encoding candidate taxa as taxonomic-lineage text and ranking image–text similarity.

Intended use & limitations

Intended for plankton image classification and retrieval over imagery similar to the thirteen imaging systems consolidated in Planktonzilla-17M. Only ~6% of plankton samples carry a species-level label and ~30% reach genus, so predictions should be read at the deepest valid taxonomic rank rather than assumed species-level. Out-of-distribution imagers and long-tail taxa degrade performance; not a sole basis for ecological decisions without expert review.

Citation

If you use Planktonzilla in your research, please cite as:

A. G. Contreras Montanares, L. Valenzuela, L. Martí, and N. Sanchez‑Pi, Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems, Inria Chile Research Center, Tech. Rep., May 2026, doi: 10.48550/arXiv.2606.00080, arXiv: 2606.00080 [cs.CV]. url: https://arxiv.org/abs/2606.00080

@techreport{contrerasmontanares:hal-05621003,
  title         = {Planktonzilla: {M}ultimodal dataset and models for understanding plankton ecosystems},
  author        = {Contreras Montanares, Alan Gerson and Valenzuela, Luis and Mart{\'i}, Luis and Sanchez-Pi, Nayat},
  year          = 2026,
  month         = {May},
  keywords      = {Explainable AI; XAI ; Plankton Classification ; CLIPS ; Multimodal Classification},
  eprinttype    = {arxiv},
  hal_id        = {hal-05621003},
  hal_version   = {v1},
  eprint        = {2606.00080},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.00080},
  doi           = {10.48550/arXiv.2606.00080},
  institution   = {Inria Chile Research Center},
}

Model produced by Project OcéanIA / Inria Chile Research Center. Code: Inria-Chile/planktonzilla.