--- license: cc-by-4.0 tags: - medical-imaging - radiology - 3d - object-detection - nndetection - pretrained-backbone - supervised --- ## ResEncL-MissingPiece-MultiTalent Copyright German Cancer Research Center (DKFZ) and contributors. Please make sure that your usage of these models is in compliance with their license. [![MICCAI 2025](https://img.shields.io/badge/MICCAI-2025-1f6feb.svg)](https://doi.org/10.1007/978-3-032-04965-0_58) `ResEncL-MissingPiece-MultiTalent` is a ResEnc-L backbone for **3D medical object detection**, pre-trained with **MultiTalent** supervised multi-dataset segmentation on 65 public segmentation datasets of mixed modalities. It is one of the checkpoints released with *[The Missing Piece: A Case for Pre-training in 3D Medical Object Detection](https://doi.org/10.1007/978-3-032-04965-0_58)* (MICCAI 2025). ## Model family All checkpoints of the [collection](https://huggingface.co/collections/MIC-DKFZ/the-missing-piece-pre-trained-nndetection-backbones-6ab62e4d50b7219ef37d1d50): | Model | Pre-training | Architecture | Repository | |---|---|---|---| | ResEncL-MissingPiece-MAE | [MAE](https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html) (self-supervised) | [ResEnc-L](https://arxiv.org/abs/2404.09556) | [MIC-DKFZ/ResEncL-MissingPiece-MAE](https://huggingface.co/MIC-DKFZ/ResEncL-MissingPiece-MAE) | | ResEncL-MissingPiece-MG | [Models Genesis](https://doi.org/10.1016/j.media.2020.101840) (self-supervised) | [ResEnc-L](https://arxiv.org/abs/2404.09556) | [MIC-DKFZ/ResEncL-MissingPiece-MG](https://huggingface.co/MIC-DKFZ/ResEncL-MissingPiece-MG) | | ResEncL-MissingPiece-S3D | [Spark 3D](https://arxiv.org/abs/2410.23132) (self-supervised) | [ResEnc-L](https://arxiv.org/abs/2404.09556) | [MIC-DKFZ/ResEncL-MissingPiece-S3D](https://huggingface.co/MIC-DKFZ/ResEncL-MissingPiece-S3D) | | ResEncL-MissingPiece-VoCo | [VoCo](https://openaccess.thecvf.com/content/CVPR2024/html/Wu_VoCo_A_Simple-yet-Effective_Volume_Contrastive_Learning_Framework_for_3D_Medical_CVPR_2024_paper.html) (self-supervised) | [ResEnc-L](https://arxiv.org/abs/2404.09556) | [MIC-DKFZ/ResEncL-MissingPiece-VoCo](https://huggingface.co/MIC-DKFZ/ResEncL-MissingPiece-VoCo) | | **ResEncL-MissingPiece-MultiTalent** | [MultiTalent](https://doi.org/10.1007/978-3-031-43898-1_62) (supervised) | [ResEnc-L](https://arxiv.org/abs/2404.09556) | this repository | | RetinaUNet-MissingPiece-MultiTalent | [MultiTalent](https://doi.org/10.1007/978-3-031-43898-1_62) (supervised) | Retina U-Net (nnDetection `ConvBackbone` + FPN) | [MIC-DKFZ/RetinaUNet-MissingPiece-MultiTalent](https://huggingface.co/MIC-DKFZ/RetinaUNet-MissingPiece-MultiTalent) | The **nnFoundation** models [nnFoundationCNN](https://huggingface.co/MIC-DKFZ/nnFoundationCNN) (ResEnc-L) and [nnFoundationViT](https://huggingface.co/MIC-DKFZ/nnFoundationViT) (Primus) can be finetuned for detection with nnDetection in the same way -- see the [finetuning docs](https://github.com/MIC-DKFZ/nnDetection/blob/main/docs/finetuning.md). ## Using this checkpoint Fine-tune it for detection with **[nnDetection -- Finetuning pretrained backbones](https://github.com/MIC-DKFZ/nnDetection/blob/main/docs/finetuning.md)**: ```bash hf download MIC-DKFZ/ResEncL-MissingPiece-MultiTalent checkpoint_final.pth --local-dir ./checkpoints/ResEncL-MissingPiece-MultiTalent nndet_train Task_YourDataset residual_encoder_retinaunet_focal_v002 0 \ -o module=RetinaUNetFocalV002_ResEnc_TL exp.tag=_MultiTalent \ +transfer_learning_ckpt=./checkpoints/ResEncL-MissingPiece-MultiTalent/checkpoint_final.pth \ --transfer_learning --load_adapt_plan ``` `--load_adapt_plan` rebuilds the backbone to match this checkpoint before loading the weights. For a **Deformable DETR** head instead, use `residual_encoder_def_detr_v002` with `-o module=BoxDeformableDETRV002_ResEnc_TL`. ## Repository contents | File | Purpose | |---|---| | `checkpoint_final.pth` | the pre-trained weights | | `adaptation_plan.json` | architecture + preprocessing plan; nnDetection reads it (from the checkpoint) to rebuild the backbone | | `config.json` | placeholder so the Hub records download counts | ## Expected input 3D volumes, preprocessed with nnDetection's standard pipeline (`nndet_prep`), using the dataset's **default planned spacing**. Downstream patch size: **128 × 128 × 128**. For reference, pre-training used Z-score normalised volumes resampled to 1 × 1 × 1 mm, with a patch size of 192 × 192 × 192. The checkpoint contains **87 input stems**, one per pre-training (sub-)dataset (some of the 65 datasets were split into several stems). The adaptation plan selects the CT stem (`encoder.stem.351_0`) by default; to use a different one, pass `-o model_cfg.stem_override=encoder.stem.` (see [§3.4 of the finetuning docs](https://github.com/MIC-DKFZ/nnDetection/blob/main/docs/finetuning.md)). ## Checkpoint format `checkpoint_final.pth` is a `torch.save` dictionary that loads safely with `weights_only=True`: | Key | Contents | |---|---| | `network_weights` | the pre-trained `state_dict` (only the encoder and input stem are transferred downstream) | | `trainer_name` | the pre-training trainer | | `nnssl_adaptation_plan` | same content as `adaptation_plan.json` | | `citations` | the references to cite when using these weights (printed to the training log on load) | > **Note:** this checkpoint stores a number of `state_dict` entries as aliases of the same > underlying tensor (the `all_modules.*` keys mirror the named conv/norm modules). This is > expected for ResEnc -- do not deduplicate these keys. ## Citation If you use these weights, please cite **The Missing Piece** (MICCAI 2025):
The Missing Piece BibTeX ```bibtex @inproceedings{eckstein2025missing, title = {The Missing Piece: A Case for Pre-training in 3D Medical Object Detection}, author = {Eckstein, Katharina and Ulrich, Constantin and Baumgartner, Michael and K{\"a}chele, Jessica and Bounias, Dimitrios and Wald, Tassilo and Floca, Ralf and Maier-Hein, Klaus H.}, booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2025}, series = {Lecture Notes in Computer Science}, volume = {15963}, pages = {615--626}, year = {2025}, publisher = {Springer Nature Switzerland}, doi = {10.1007/978-3-032-04965-0_58} } ```
Please also cite the architecture, pre-training method and pre-training data behind this checkpoint:
Further references **Architecture -- ResEncL** - Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., & Jaeger, P. F. (2024). nnU-Net revisited: A call for rigorous validation in 3D medical image segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 (Lecture Notes in Computer Science, pp. 488–498). Springer. https://doi.org/10.1007/978-3-031-72114-4_47 - Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z **Pretraining Method -- MultiTalent** - Ulrich, C., Isensee, F., Wald, T., Zenk, M., Baumgartner, M., & Maier-Hein, K. H. (2023). MultiTalent: A multi-dataset approach to medical image segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 (Lecture Notes in Computer Science, Vol. 14222, pp. 648–658). Springer. https://doi.org/10.1007/978-3-031-43898-1_62