license: mit
tags:
- connectome
- drosophila
- computational-neuroscience
- drone
- fpv
- liftoff
- pytorch
- recurrent-neural-network
- imitation-learning
- reinforcement-learning
library_name: pytorch
pipeline_tag: reinforcement-learning
Haltere: a fruit-fly connectome brain that flies an FPV drone
A recurrent network whose 30,000 neurons and 2.77 million synaptic connections are copied from the Janelia male CNS connectome v1.0 (FlyEM, Cambridge Connectomics, Google Connectomics), trained to fly a quadcopter in the FPV simulator Liftoff. The drone's senses are written into the fly's own sensory neurons (haltere, wing campaniform, optic-flow, ocellar, Johnston's organ, compass and goal cells) and the four stick commands are read out of the wing motor neurons and their premotor partners. Synaptic structure and sign are fixed by the connectome; per-synapse gains, neuron gains, biases, time constants, sensory encoders and the readout are trained.
Code, training pipeline, Liftoff integration and videos: https://github.com/skulitom/haltere
Files
| file | what | use |
|---|---|---|
ftSmooth_best.pt |
recommended brain for Liftoff: fine-tuned with 50 ms extra latency and a smoothness penalty | haltere liftoff fly ftSmooth_best.pt |
ftRobust_best.pt |
wide domain randomization; first brain that flew in Liftoff | |
imJ_best.pt |
imitation of the MLP controller with the premotor readout | |
mlp_baseline.pt |
the MLP teacher (no connectome) | control experiment |
flight.npz, flight.nodes.parquet, flight.meta.json |
the built flight graph: 30,000 neurons, signed synapse counts, named populations | required by every brain checkpoint |
The checkpoints are slim (parameters only, about 12 MB); the graph is loaded next to them. Full checkpoints with optimizer state are on the GitHub release.
Results
| controller | simulator, full difficulty | Liftoff |
|---|---|---|
| MLP baseline | 0.05 m mean error, 100% within 0.5 m | not flown |
imJ_best (imitation) |
0.22 m, 95% | drifts 1.4 m on the physics stand-in |
ftRobust_best (+ domain randomization) |
0.20 m, 99.6% | 2 m hover, 0.34 m mean error over 40 s; 3 m square pattern |
ftSmooth_best (+ latency, smoothness) |
0.30 m, 95% (50 ms delay) | 4x smaller stick jitter at Liftoff-like latency |
Full difficulty: 25 degrees of tilt, 90 deg/s rotation, 1 m/s velocity and 1 m offset at the start, targets anywhere in a 6 x 6 x 2 m box, physics jittered by 35%.
How to use
git clone https://github.com/skulitom/haltere && cd haltere
uv venv --python 3.13 .venv && uv pip install --python .venv/Scripts/python.exe torch --index-url https://download.pytorch.org/whl/cu128
uv pip install --python .venv/Scripts/python.exe -e ".[dev]"
# put the files of this repo into artifacts/ and data/built/, then:
haltere eval artifacts/ftSmooth_best.pt # simulator evaluation
haltere render artifacts/ftSmooth_best.pt # neural activity + drone video
haltere liftoff doctor # everything needed to fly it in Liftoff
Data
Connectome: male CNS v1.0, Janelia FlyEM, CC BY 4.0, downloaded by haltere fetch from
gs://flyem-male-cns (neuron annotations, neurotransmitter predictions, connection weights). Not
redistributed here.
Method in one paragraph
Lappalainen et al. 2024-style connectome-constrained RNN (rate units, weights proportional to synapse counts with fixed neurotransmitter signs) on a 30k-neuron subgraph of the male CNS (all neurons within two synapses of the flight senses and the wing motor neurons, plus the central complex and all descending neurons). Trained in a batched differentiable quadrotor simulator with Betaflight-style rates and rate PID (Liftoff's own Zetaflight gains) by imitation of an MLP controller, then fine-tuned by back-propagation through the simulator with domain randomization. Flown in Liftoff through its UDP telemetry and a virtual Xbox controller (ViGEmBus).