jaichang/embeddinggemma2-korean-birds-620

🐦 EmbeddingGemma 2 (740M) — Korean 620 Bird Species Trimodal Bird Finder (Main Model)

English | 한국어

---

🌍 English Overview

This repository hosts the fine-tuned EmbeddingGemma 2 (google/embeddinggemma-2, 740M) trimodal model and multi-vector reference index covering 620 species of wild birds. It aligns Bird Audio (16kHz vocalizations), Bird Photos (RGB field images), and Bilingual Ecological Descriptions (Korean + English taxonomy & acoustic traits) into a single unified 768-dimensional L2-normalized embedding space.

📱 Looking for the lightweight On-Device Mobile (LiteRT .tflite) & Browser (WebGPU .onnx) models? Visit the companion on-device repository: jaichang/embeddinggemma2-korean-birds-620-litert

✨ Key Highlights

• Unified 768-D Trimodal Space: Query by bird call (.wav/.mp3), field photo (.jpg/.png), or natural-language description (English/Korean) and retrieve the Top-$K$ matching bird species via simple cosine similarity (dot product).
• Fine-Tuned on Real Wildlife Observations: Trained with Multiple Negatives Ranking Loss (InfoNCE) across audio, visual, and encyclopedic text pairs sourced from iNaturalist Research-Grade observations and Wikipedia.
• Teacher Model for On-Device Knowledge Distillation: Serves as the 768-d teacher space for our ultra-compact mobile/WebGPU student encoders (9.2 MB Vision .onnx & 14.7 MB Audio .onnx in jaichang/embeddinggemma2-korean-birds-620-litert).

📦 Files in This Repository

File	Description
model.safetensors, config.json, tokenizer.json	Fine-tuned EmbeddingGemma 2 (740M) weights and processor configuration for Python / PyTorch / SentenceTransformers server & Colab inference
korean_620_birds_multivector_index.npz	Multi-vector NumPy archive containing audio_hybrid_embs, photo_hybrid_embs, and text_embs (float32, shape [N, 768])
korean_620_birds_catalog.json	Full metadata catalog for 620 species (Korean/English/Scientific names, descriptions, reference image URLs, and audio URLs)
korean_birds_gallery_grid.png, korean_620_birds_result_grid.png, korean_620_birds_photo_search_grid.png	Evaluation & retrieval visualization grids

🚀 Quick Start (Python)

import json
import numpy as np
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer

repo_id = "jaichang/embeddinggemma2-korean-birds-620"
model = SentenceTransformer(repo_id, trust_remote_code=True)

# Load precomputed 768-d reference index & catalog
npz_path = hf_hub_download(repo_id=repo_id, filename="korean_620_birds_multivector_index.npz")
cat_path = hf_hub_download(repo_id=repo_id, filename="korean_620_birds_catalog.json")

index = np.load(npz_path)
catalog = json.load(open(cat_path, encoding="utf-8"))["species"]

# Text-to-Bird search example (works in both English and Korean)
query = "black and white bird with long tail and iridescent blue-green wings, chak-chak call"
q_emb = model.encode(query, normalize_embeddings=True)

sims = index["photo_hybrid_embs"] @ q_emb
top1 = int(np.argmax(sims))
print(f"Top-1: {catalog[top1]['en_name']} ({catalog[top1]['kr_name']}, {catalog[top1]['sci_name']}) — cos={sims[top1]:.4f}")

📚 Dataset Provenance & Attribution

Modality	Source & API	Details & License
Taxonomy	iNaturalist Open Taxonomy API & Korea Biodiversity Information System (KBR)	All 620 native & migratory bird species observed in Korea (place_id=6891, IDs 1–620)
Audio (16kHz)	iNaturalist Open Data Audio Archive (sounds=true&quality_grade=research)	4–5 sec vocalization clips (CC0 / CC-BY per observation)
Vision (RGB)	iNaturalist Open Data S3 Photo Archive (inaturalist-open-data.s3.amazonaws.com)	Reference taxonomy photos & unseen wild test photos (CC0 / CC-BY)
Text (KO/EN)	Korean & English Wikipedia REST API (api/rest_v1/page/summary/)	Combined habitat, plumage, and vocalization encyclopedia entries (CC BY-SA 4.0)

---

🇰🇷 한국어 안내 (Korean Overview)

야생조류 620종 전체의 새소리(Audio), 새 사진(Vision), 생태 도감(Text)을 단일 768차원 임베딩 공간에서 검색하는 본 모델(PyTorch / SentenceTransformers) 레포지토리입니다.

📱 Flutter 모바일 앱(LiteRT .tflite) 및 웹 브라우저(WebGPU .onnx) 온디바이스 추론 전용 레포지토리는 아래 링크에 분리되어 있습니다: 👉 jaichang/embeddinggemma2-korean-birds-620-litert

✨ 주요 특징

• 단일 768차원 트라이모달(Trimodal) 정렬: 새소리 녹음, 야생 사진, 자연어 특징 설명(한국어/영어)을 모두 동일한 768차원 초구면(L2-normalized) 공간에 매핑하여 코사인 유사도 내적만으로 즉시 종을 판별합니다.
• 온디바이스 지식증류(Knowledge Distillation) 교사 모델: 본 모델(740M, 1.2GB)이 생성한 768차원 임베딩 공간을 정답지(Teacher)로 삼아, 모바일/웹에서 1초 미만에 동작하는 초경량 학생 인코더(9.2MB 비전 / 14.7MB 오디오)를 지식증류하는 기반 모델입니다.

📚 데이터셋 출처 및 라이선스

모달리티	출처 (Source & API)	상세 내역 및 라이선스
조류 분류 체계	iNaturalist Open Taxonomy API & 국가생물다양성정보공유체계(KBR)	한국 620종(place_id=6891, 1620번) + 글로벌 확장 조류 1,380종(iconic_taxa=Aves, 6212,000번)
새소리 오디오 (Audio)	iNaturalist Open Data Audio Archive (sounds=true&quality_grade=research)	16kHz 오디오 구간 추출, 각 클립별 CC0 / CC-BY 라이선스 명시
야생조류 사진 (Vision)	iNaturalist Open Data S3 Photo Archive (inaturalist-open-data.s3.amazonaws.com)	도감 대표 사진 + 미학습 야생 테스트 사진 (CC0 / CC-BY)
생태·음향 도감 설명 (Text)	한국어/영어 위키백과 REST API (ko.wikipedia.org/api/rest_v1/page/summary/)	국명·영문명·학명·서식지·울음소리·외형 결합 도감 문서 (CC BY-SA 4.0)