README.md

metadata

title: DESI Spectra Zoo
emoji: 🧬
colorFrom: yellow
colorTo: red
sdk: docker
app_file: app.py
pinned: false
short_description: Expert annotation of DESI spectra

DESI Spectra Zoo

Classify DESI spectra one-by-one to build a Gold dataset of labelled astronomical objects (Galaxy, Star, QSO, Interesting, Bad Data).

Built on MultimodalUniverse/desi (~100K spectra).

Environment variables

Local development

Create a .env file in the project root (never commit this):

# HuggingFace — required for persistent annotation state across restarts
HF_TOKEN=hf_...
HF_LOG_REPO_ID=your-username/desi-spectra-zoo-logs

# Optional
HF_LOG_EVERY_MINUTES=10

# Dataset
DATASET_ID=MultimodalUniverse/desi
DATASET_CONFIG=default
DATASET_SPLIT=train
ID_COLUMN=object_id
SPECTRUM_COLUMN=spectrum
POOL_SIZE=500
POOL_SEED=42

# Plot cache
PLOT_CACHE_DIR=cache/plots
PLOT_SMOOTH=3

Then run:

python app.py

The app serves at http://localhost:7860.

HuggingFace Spaces

Set variables in Settings → Variables and secrets for your Space:

Variable	Type	Description
HF_TOKEN	Secret	HuggingFace token with read/write access to HF_LOG_REPO_ID
HF_LOG_REPO_ID	Secret	Dataset repo for persisting annotation state, e.g. your-username/desi-spectra-zoo-logs
HF_LOG_EVERY_MINUTES	Variable	How often to sync state to HF Hub (default: 10)
DATASET_ID	Variable	HF dataset to load spectra from (default: MultimodalUniverse/desi)
DATASET_CONFIG	Variable	Dataset config name (default: default)
DATASET_SPLIT	Variable	Dataset split (default: train)
ID_COLUMN	Variable	Name of the ID column (default: object_id)
SPECTRUM_COLUMN	Variable	Name of the spectrum column (default: spectrum)
POOL_SIZE	Variable	Number of spectra to pre-load (default: 500)
POOL_SEED	Variable	Fixed seed so all annotators see the same pool (default: 42)
PLOT_SMOOTH	Variable	Smoothing kernel size for spectrum plots (default: 3)

Note: HF_TOKEN must be added as a Secret (not a variable) to keep it private.

Persistent annotation state

Annotations are written to state/annotations.json on every classification and periodically synced to HF Hub via CommitScheduler. Annotation events are also logged as JSONL in data/.

Without HF_LOG_REPO_ID set, state is only saved locally and will be lost on container restart.

How it works

1. On startup, metadata for the first 500 spectra is streamed (fast, no spectrum data).
2. The first spectrum plot is generated synchronously — app becomes available.
3. In the background: the full 100K metadata index loads, then remaining pool plots are generated.
4. Filters (redshift, E(B-V), ZWARN) operate on the full index once loaded.
5. When filters select spectra outside the initial pool, their data is fetched on-demand.

License

MIT