source/PROGRESS.md

Fly XOX — resume here

Last updated: 2026-09-14 07:45 UTC. Training and WGSL validation completed; private artifact upload and cleanup pending.

Latest result (supersedes older pending notes below)

• Training completed: 1,200 updates, ~42.8 seconds reported training-loop duration. Best validation checkpoint at step 1,100.
• Training optimal-move accuracy: 98.17% on 3,560 positions.
• Validation optimal-move accuracy: 90.04% on 492 positions.
• Held-out test: 91.67% on 468 positions, versus 62.82% for the untrained same model and 56.91% uniform-random expected optimal-move rate.
• Fixed input/output interfaces unchanged. Internal parameters changed: 153,090 gains, 165,455 biases, 165,455 leaks.
• 500,100 trainable internal parameters; ~1.01 GB peak GPU allocation. One seed only, no biological/superiority claim.
• Tool installation completed. train/evaluate_games.py, train/export.py, train/verify_webgpu.py have been launched sequentially remotely; inspect their process/output before relaunching.
• Source train/verify_webgpu.py and all web files uploaded. Local source remains dependency/data free.
• Complete-game evaluation (500 games each): vs random, fly first 474 W / 26 D / 0 L, fly second 425 W / 70 D / 5 L; vs minimax, fly first 0 W / 500 D / 0 L, fly second 0 W / 480 D / 20 L. Game trajectories can include training positions; this is separate from the held-out-position score.
• Export completed: 224,749,428 bytes (~225 MB), retaining every edge, 16 MiB chunks.
• Exact browser WGSL passed wgpu-native verification on all 12 golden boards; all moves match PyTorch, max absolute logit difference 0.00000500679.
• Native validation adapter is software Vulkan / llvmpipe (CPU), not the RTX GPU. Median ~762 ms is a software-backend timing, NOT expected browser GPU latency. Browser/device performance remains unmeasured.
• Remote apt installed old Node 12, which rejected valid optional chaining during syntax checks. Obtain a current Node binary on the REMOTE host only, then rerun JS checks. Do not rewrite modern browser code merely to accommodate old Node.

User's requirements and authorization

• Build a tic-tac-toe (XOX) player using the actual MaleCNS fly connectome, inspired by the September 2026 fly-brain projects.
• Train internal model parameters, rather than only putting a normal game AI behind a fly graphic.
• Train on Vast.ai. USD 5 total pilot ceiling, including compute, storage, transfer.
• Author source locally in /Users/n4ze3m/personal/fun/fly-xox.
• Do not install dependencies, download datasets/checkpoints, or run training locally. All dependencies, model files, preparation, tests, training, and exports belong on the remote machine.
• Finished browser application should execute the model via WebGPU.
• Keep model and checkpoints in a private Hugging Face repository. User explicitly chose private. Do not make the repository or artifacts public.
• User asked for this progress file because the session may end soon.

Active infrastructure — CHECK FIRST ON RESUME

• Vast.ai instance: 50989186, label fly-xox-pilot-5usd.
• GPU: RTX 3090, 24 GiB VRAM; about 43 GB available host RAM; 30 GB rented disk.
• Offer 30239913, quoted total $0.1388888889/hour including disk. Disk component $0.0055555556/hour continues when stopped.
• Container: pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime.
• Direct SSH: ssh -p 44245 root@77.48.24.240.
• Proxy SSH: ssh -p 29186 root@ssh8.vast.ai.
• Existing local ~/.ssh/id_ed25519 is registered. Do not print private key material.
• Remote project: /workspace/fly-xox.
• Port 8080 is mapped to host port 44232; do NOT use public unauthenticated serving for private model assets.
• Vast CLI is already installed at /Users/n4ze3m/.local/bin/vastai and authenticated. Use --raw, and filter account output to avoid printing identity/credentials.
• Credit before creating this instance: $9.1766269051. No other instances were present. Budget floor for this pilot: $4.1766269051, with a safety margin above that.
• A local detached fallback command was started near creation: sleep 10800; vastai stop instance 50989186 --raw. It is only a fallback (local sleep/offline can delay it), not a durable cloud budget cap. Stop compute yourself when done. Do not assume stopping deletes disk charges.
• Do not destroy the instance until remote artifacts have been verified on Hugging Face; then destroy it to stop all charges.

Hugging Face

• Existing local hf CLI and authenticated account n4ze3m; hf auth whoami successfully refreshed existing OAuth credentials.
• Existing token file: ~/.cache/huggingface/token; never print it or store it in project files.
• Created private model repository: https://huggingface.co/n4ze3m/fly-xox-malecns.
• At this checkpoint it has no uploaded model artifacts yet.
• remote/publish_hf.py uploads directly from Vast, requires a token on stdin only, and refuses if the model repository isn't private.
• If uploading via SSH, pipe the existing refreshed token to the remote process stdin. Never include it in command arguments, logs, Git, or persistent remote files. No dataset/checkpoint bytes should pass through local files.

Completed so far

• Read Vast.ai skill and relevant Sites guidance. User's no-local-install/download rule overrides normal local scaffolding/install steps. Native HTML/JS/WGSL source has no browser package dependencies.
• Created source under fly-xox/, with task constraints also in AGENTS.md.
• Remote dependency installation succeeded: PyTorch 2.8.0+cu128, numpy 2.2.6, scipy 1.15.3, pandas 2.2.3, pyarrow 20.0.0, wgpu 0.24.0.
• Downloaded three official MaleCNS v1.0 Feather files on Vast only. Checked upstream MD5 when supplied and recorded SHA256 source lock.
• Prepared graph: 166,700 neurons, 25,582,938 directed neuron-pair edges, 124,177,617 retained synaptic contacts, 17,937 annotated sensory neurons.
• Retention: assigned nonempty superclass, excluding explicit Glia; all released connections between retained IDs; no extra edge threshold.
• Remote python train/test_core.py: 3 tests passed (exact game solver, 4,520 nonterminal reachable positions with symmetry-grouped split/legality, sparse forward and state gradients vs dense reference).
• Initial pilot hit a float64 readout vs float32 state mismatch; fixed by casting the normalized readout to float32 in train/model.py, uploaded fix, and restarted.
• Remote installation of Vulkan loader/Mesa/nodejs and huggingface_hub was launched separately for WebGPU validation and upload. Check its log for completion.

Training design

• Full retained sparse graph is present for every forward pass. No normal-policy/minimax fallback in the browser.
• Engineered symbolic board input: 27 one-hot channels (empty/self/opponent for each square), fixed random projection into annotated sensory neurons. This is not pixel vision.
• Signed, incoming-normalized contact weights. GABA/glutamate are assigned negative signs, others/unknown positive; this is a disclosed crude assumption, not receptor physiology.
• Four recurrent softsign rate updates, reset state each move. No biological time claim.
• Learn three internal parameters per neuron: incoming gain, offset, leak (500,100 trainable parameters). Graph topology and external input/output adapters stay fixed.
• Gain is 0.1 + 5.9sigmoid(raw); leak is 0.05 + 0.9sigmoid(raw).
• Readout: fixed signed random projection of 2,048 non-sensory neurons selected by response variance on training boards only; nine move scores.
• Exact minimax teacher is training/evaluation only. Loss maximizes total probability of all tied optimal legal moves.
• Train/validation/test split groups player-relative rotations/reflections by SHA256, roughly 80/10/10.
• Training invocation currently launched: python -u train/pilot.py --updates 1200 --seconds 2400, default batch 32 and LR 0.025.
• Select best checkpoint on validation only; held-out test evaluated after selection. Compare against untrained same model and uniform random optimal-move rate.

Remote logs / outputs

• Setup: /workspace/fly-xox-bootstrap.log (completed data import).
• Training: /workspace/fly-xox-training.log.
• Vulkan/node/HF tool setup: /workspace/fly-xox-tools.log.
• Graph: /workspace/fly-xox/data/graph/.
• Boards: /workspace/fly-xox/data/boards.npz.
• Checkpoints/results: /workspace/fly-xox/runs/pilot/ (untrained.pt, best.pt, last.pt, history/results when generated).
• Exports will be /workspace/fly-xox/web/model/. Do not rsync this back locally.

Source files and next actions

1. Check instance status and current training/tool logs. Ensure no duplicate training process is started.
2. train/pilot.py: bounded training, validation selection, final held-out results. Do not assert successful learning until results exist.
3. After source uploads finish, run remotely:
   • python train/evaluate_games.py — 500 random/minimax games per side using cached model predictions inside evaluator only.
   • python train/export.py — bakes gains into effective graph weights and encoder, writes 16 MiB binary chunks, manifest, 12 golden positions.
   • python train/verify_webgpu.py — runs the app's exact WGSL through wgpu-native, compares PyTorch logits/moves, records actual adapter and timing. Not browser end-to-end testing.
4. web/brain.wgsl, web/readout.wgsl: full sparse graph + fixed output projection. Seven storage bindings in brain shader; largest buffer ~102 MB, fits typical 128 MiB binding limit (verify actual device).
5. web/engine.js: streamed chunks with SHA256 checks, WebGPU device/buffer validation, identical update loop, legal-move masking. Loads ./model/ by default. Model assets must be served with authenticated access or bundled into a private host; don't leak a HF token into client code.
6. web/index.html, style.css, app.js: authored game UI, actual activity bars, legal turns, win/draw handling, model loading/error states, reset and fly-first controls. Not yet runtime-validated.
7. remote/publish_hf.py: preserve exports, checkpoints, graph, source directly on private HF after checks pass. Verify remote contents/hashes before deleting Vast disk.
8. Run JavaScript syntax checks remotely (node is being installed). Add focused frontend/state tests as needed. Do not install/test locally.
9. Finish README with actual results, reproduction, limitations, and spending. Update this file with final status and instance cleanup.

Open issues / cautions

• Successful learning has not yet been established. Pilot accuracy and WebGPU parity are pending.
• Native WebGPU initially found no adapters because libvulkan was absent. NVIDIA ICD and 610.57.04 graphics libraries are present; remote apt installation of libvulkan1/mesa-vulkan-drivers should resolve. Prefer NVIDIA adapter; report if only software Vulkan is available.
• HF is private by explicit choice. Do not silently expose model chunks on public HTTP, public Spaces, or public Sites.
• A private static Hugging Face Space is a possible way to host the game without an embedded token or paid inference server. Official docs say Static Spaces are free and support private access. This has NOT been created or promised yet; user selected HF as artifact storage, not explicitly a Space.
• No local node_modules, graph/model/checkpoint downloads, training runs, build outputs, or Python environments were created. Local SSH known-hosts entries and existing HF OAuth token refresh are normal connection metadata.
• Browser play itself eventually fetches model bytes into the visitor's memory; user was informed of this distinction. Do not open the game on their computer automatically during this no-download development phase.