CLIP-ViT-B-16.bioclip-pt.planktonzilla-pt

CLIP-style (image–text) plankton model from the paper Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems (Contreras, Valenzuela, Martí & Sanchez-Pi, 2026). It is one arm of the paper's controlled comparison between supervised and CLIP-style training on a shared ViT backbone, trained with taxonomic lineage used as the text description (following the BioCLIP recipe).

• Architecture: CLIP ViT-B/16 (embed dim 512, 224×224 input)
• Initialised from: BioCLIP (BioCLIP)
• Trained on: the 3.74M-image plankton subset of Planktonzilla-17M (60/20/20 train/val/test, stratified by source dataset and taxonomic label)
• Recipe: OpenCLIP, 100 epochs, global batch 16,384, 64× H100; taxonomic-lineage captions
• Preprocessing: CLIP mean/std, bicubic, shortest-side resize

Results

Macro-F1 per taxonomic rank on the in-domain test split (Planktonzilla-17M plankton subset), from Table 2 of the paper. Models are selected on the highest validation Macro-F1.

Kingdom	Phylum	Class	Order	Family	Genus	Species
0.932	0.858	0.810	0.800	0.768	0.752	0.734

The paper reports that models trained on Planktonzilla-17M substantially outperform off-the-shelf BioCLIP / BioCLIP 2 across all taxonomic ranks, and that fully supervised classifiers on the same backbone match or slightly exceed these CLIP-style models.

Usage (open_clip)

import open_clip

hub = "hf-hub:project-oceania/CLIP-ViT-B-16.bioclip-pt.planktonzilla-pt"
model, _, preprocess = open_clip.create_model_and_transforms(hub)
tokenizer = open_clip.get_tokenizer(hub)

Use it for zero-/few-shot classification by encoding candidate taxa as taxonomic-lineage text and ranking image–text similarity.

Intended use & limitations

Intended for plankton image classification and retrieval over imagery similar to the thirteen imaging systems consolidated in Planktonzilla-17M. Only ~6% of plankton samples carry a species-level label and ~30% reach genus, so predictions should be read at the deepest valid taxonomic rank rather than assumed species-level. Out-of-distribution imagers and long-tail taxa degrade performance; not a sole basis for ecological decisions without expert review.

Citation

@article{contreras2026planktonzilla,
  title   = {Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems},
  author  = {Contreras, Alan and Valenzuela, Luis and Mart\'i, Luis and Sanchez-Pi, Nayat},
  journal = {arXiv preprint arXiv:2606.00080},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.00080}
}

Model produced by Project OcéanIA / Inria Chile Research Center. Code: Inria-Chile/planktonzilla.