Gregory-L commited on
Commit
0ab4e0b
Β·
verified Β·
1 Parent(s): d1c88ba

Your own bankML in one command: ./install.sh --space (github main a27c4b6)

Browse files
Files changed (6) hide show
  1. CHANGELOG.md +2 -1
  2. docs/install.md +1 -0
  3. docs/usage.md +5 -3
  4. index.html +6 -4
  5. install.sh +21 -2
  6. sAGI/models.py +12 -0
CHANGELOG.md CHANGED
@@ -31,7 +31,8 @@ decode speed, the third, is measured on an idle machine next.
31
  - `bankml serve … --allow-origin ORIGIN` lets one named web page call the gateway from a browser: its CORS preflight
32
  is answered (with Chrome's private-network grant, as a public page reaching a loopback address requires) and its
33
  answers, streamed or not, carry `Access-Control-Allow-Origin`; any other origin gets none and a 403 preflight. The
34
- loopback `Host` and JSON-POST rules are unchanged. It is what lets the bankML Space's page
 
35
  ([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml)) talk to a visitor's own bankML, free, on their own
36
  CPU β€” the way Savante's page reaches a local engine. Checked live: the preflight, plain and streamed answers with
37
  the receipt, and another origin refused.
 
31
  - `bankml serve … --allow-origin ORIGIN` lets one named web page call the gateway from a browser: its CORS preflight
32
  is answered (with Chrome's private-network grant, as a public page reaching a loopback address requires) and its
33
  answers, streamed or not, carry `Access-Control-Allow-Origin`; any other origin gets none and a 403 preflight. The
34
+ loopback `Host` and JSON-POST rules are unchanged. `./install.sh start --space` turns it on for the Space (remembered
35
+ in `install.env`, applied to the carrier `sAGI/models.py` starts; `--no-space` turns it off). It is what lets the bankML Space's page
36
  ([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml)) talk to a visitor's own bankML, free, on their own
37
  CPU β€” the way Savante's page reaches a local engine. Checked live: the preflight, plain and streamed answers with
38
  the receipt, and another origin refused.
docs/install.md CHANGED
@@ -135,6 +135,7 @@ alone brings the whole stack back. Use `./install.sh model start` to wait for th
135
  | `--no-start` | the default run ends after `model` (or `voice`): everything installed, Savante not started. `bankml serve` is still started by `model` |
136
  | `--view` | `start` also starts view mode on `0.0.0.0:7874`. It works with named steps too: `./install.sh start --view` |
137
  | `--voice` | the default run includes `voice`. With named steps, name it instead: `./install.sh voice` |
 
138
  | `-h`, `--help` | prints the header of `install.sh` and exits |
139
 
140
  Any other argument stops the installer with exit code 2.
 
135
  | `--no-start` | the default run ends after `model` (or `voice`): everything installed, Savante not started. `bankml serve` is still started by `model` |
136
  | `--view` | `start` also starts view mode on `0.0.0.0:7874`. It works with named steps too: `./install.sh start --view` |
137
  | `--voice` | the default run includes `voice`. With named steps, name it instead: `./install.sh voice` |
138
+ | `--space` / `--no-space` | 0.3.9: let bankML's Hugging Face page ([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml)) talk to your `bankml serve` from your browser β€” its *Your own bankML* mode: your CPU, your verified model, a receipt the page checks. Serve starts with `--allow-origin https://pythai-bankml.static.hf.space` (that one page, nothing else); the choice is remembered in `install.env` (`INSTALL_ALLOW_ORIGIN`; `BANKML_ALLOW_ORIGIN` overrides it). `./install.sh start --space` applies it at once (the running services stop first); `--no-space` forgets it. `./install.sh status` says which page is allowed |
139
  | `-h`, `--help` | prints the header of `install.sh` and exits |
140
 
141
  Any other argument stops the installer with exit code 2.
docs/usage.md CHANGED
@@ -247,9 +247,11 @@ In the next release (not yet tagged); each is checked against llama-server b1119
247
  llama-server `-np 1`. Like llama-server, it keeps the states of other conversations in RAM (`BANKML_CACHE_RAM`, MiB;
248
  0 off, -1 no limit; unset, 8192 MiB but at most a quarter of the memory available at load), so a conversation that comes back after
249
  another does not recompute its whole history.
250
- - **Your bankML from a web page (0.3.9).** `--allow-origin https://pythai-bankml.static.hf.space` lets the bankML
251
- Space's page talk to your own `bankml serve` from your browser: your CPU, your verified model, a receipt on every
252
- answer, nothing sent anywhere else. Only that one origin is answered with CORS headers.
 
 
253
  - **A smaller conversation memory (0.3.9, in progress).** `BANKML_CACHE_TYPE=q8_0` keeps the KV cache in q8_0 instead of f16,
254
  about half the memory, so a long context fits on a small machine. It is llama.cpp's `--cache-type-k q8_0
255
  --cache-type-v q8_0` exactly, with the Hadamard rotation llama.cpp applies around a quantized cache, and gives the
 
247
  llama-server `-np 1`. Like llama-server, it keeps the states of other conversations in RAM (`BANKML_CACHE_RAM`, MiB;
248
  0 off, -1 no limit; unset, 8192 MiB but at most a quarter of the memory available at load), so a conversation that comes back after
249
  another does not recompute its whole history.
250
+ - **Your bankML from a web page (0.3.9).** `./install.sh start --space` lets the bankML Space's page
251
+ ([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml), *Your own bankML*) talk to your own `bankml serve`
252
+ from your browser: your CPU, your verified model, a receipt on every answer, nothing sent anywhere else. It starts
253
+ serve with `--allow-origin https://pythai-bankml.static.hf.space`, so only that one page gets CORS headers;
254
+ `./install.sh start --no-space` takes it back.
255
  - **A smaller conversation memory (0.3.9, in progress).** `BANKML_CACHE_TYPE=q8_0` keeps the KV cache in q8_0 instead of f16,
256
  about half the memory, so a long context fits on a small machine. It is llama.cpp's `--cache-type-k q8_0
257
  --cache-type-v q8_0` exactly, with the Hadamard rotation llama.cpp applies around a quantized cache, and gives the
index.html CHANGED
@@ -184,10 +184,12 @@
184
  <details>
185
  <summary>Start your own bankML for this page (Linux, AVX2; free)</summary>
186
  <ol>
187
- <li><code>git clone https://github.com/cryptoAGI/bankml &amp;&amp; cd bankml &amp;&amp; ./install.sh</code> β€” builds bankML and imports and verifies Bonsai-8B.</li>
188
- <li><code>./install.sh stop</code> (the installer's own serve does not allow web pages), then:<br>
189
- <code>target/release/bankml serve .models/Bonsai-8B-Q1_0.gguf --fork ~/.local/share/bankml/forks/Bonsai-8B-Q1_0.gguf.FORK.json --native --allow-origin <span id="origin"></span></code></li>
190
- <li>Press <b>connect</b>. Your browser may ask to let this page reach your machine: that is the request for your own bankML on 127.0.0.1, and nothing else.</li>
 
 
191
  </ol>
192
  </details>
193
  </div>
 
184
  <details>
185
  <summary>Start your own bankML for this page (Linux, AVX2; free)</summary>
186
  <ol>
187
+ <li><code>git clone https://github.com/cryptoAGI/bankml &amp;&amp; cd bankml &amp;&amp; ./install.sh --space</code> β€” builds bankML,
188
+ imports and verifies Bonsai-8B, and starts it so this page (<span id="origin"></span>) may talk to it. Already installed?
189
+ <code>git pull &amp;&amp; ./install.sh build start --space</code></li>
190
+ <li>Press <b>connect</b>. Your browser may ask to let this page reach devices on your local network: that is the
191
+ request for your own bankML on 127.0.0.1, and nothing else.</li>
192
+ <li><code>./install.sh start --no-space</code> takes the permission back.</li>
193
  </ol>
194
  </details>
195
  </div>
install.sh CHANGED
@@ -11,6 +11,10 @@
11
  # --no-start install everything, start nothing
12
  # --view also start view mode for the LAN (0.0.0.0:7874)
13
  # --voice also install Savante's Piper voice (step `voice`)
 
 
 
 
14
  # power [--remove] opt-in, sudo: let your user read the CPU package energy counter (RAPL), so bankML can measure
15
  # watts and joules per token; it asks first (PLATYPUS, CVE-2020-8694), and --remove undoes it
16
  # -h, --help this text
@@ -72,7 +76,9 @@ else CANON="$HOME/cryptoAGI/savante"; fi
72
  LOGS="$DATA/logs"
73
 
74
  # ── options ──────────────────────────────────────────────────────────────────────────────────────────────────
75
- SKIP_TESTS=0 NO_START=0 WITH_VIEW=0 WITH_VOICE=0 POWER_REMOVE=0
 
 
76
  STEPS=()
77
  for a in "$@"; do
78
  case "$a" in
@@ -81,7 +87,9 @@ for a in "$@"; do
81
  --view) WITH_VIEW=1;;
82
  --voice) WITH_VOICE=1;;
83
  --remove) POWER_REMOVE=1;;
84
- -h|--help) sed -n '3,19p' "$HERE/install.sh" | sed 's/^# \{0,1\}//'; exit 0;;
 
 
85
  check|build|engine|python|canon|model|voice|start|stop|status|power) STEPS+=("$a");;
86
  *) bm_die "unknown argument: $a (see ./install.sh --help)" 2;;
87
  esac
@@ -103,6 +111,13 @@ save_env() { # save_env KEY VALUE: remember a choice in $ENV_FILE
103
  }
104
  # shellcheck disable=SC1090
105
  [ -f "$ENV_FILE" ] && . "$ENV_FILE"
 
 
 
 
 
 
 
106
  sha256_of() { if have sha256sum; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }
107
  fetch() { # fetch URL FILE: resumable download
108
  local bar=--silent; [ -t 2 ] && bar=--progress-bar
@@ -281,6 +296,9 @@ step_start() {
281
  bm_sub "talk: $(bm_link http://127.0.0.1:7873) (this computer only)"
282
  [ "$WITH_VIEW" = 1 ] && bm_sub "watch: http://<this computer's LAN address>:7874"
283
  bm_sub "check: curl -s 127.0.0.1:18093/bankml"
 
 
 
284
  bm_sub "docs: $(bm_link https://cryptoagi.github.io/bankml/)"
285
  }
286
 
@@ -310,6 +328,7 @@ print(" verified:", s.get("verified"), "Β·", s.get("model", "?").rsplit("/", 1
310
  PY
311
  fi
312
  [ -x "$BIN" ] && bm_info "$("$BIN" version)"
 
313
  bm_info "engine: ${LLAMA_SERVER:-not installed} Β· python: $PY"
314
  }
315
 
 
11
  # --no-start install everything, start nothing
12
  # --view also start view mode for the LAN (0.0.0.0:7874)
13
  # --voice also install Savante's Piper voice (step `voice`)
14
+ # --space let bankML's Hugging Face page (huggingface.co/spaces/PYTHAI/bankml) talk to your bankml serve from
15
+ # your browser ("Your own bankML": your CPU, your verified model, a receipt): serve starts with
16
+ # --allow-origin for that one page; remembered; `./install.sh start --space` applies it now
17
+ # --no-space forget it (serve answers this computer only, no web page)
18
  # power [--remove] opt-in, sudo: let your user read the CPU package energy counter (RAPL), so bankML can measure
19
  # watts and joules per token; it asks first (PLATYPUS, CVE-2020-8694), and --remove undoes it
20
  # -h, --help this text
 
76
  LOGS="$DATA/logs"
77
 
78
  # ── options ──────────────────────────────────────────────────────────────────────────────────────────────────
79
+ SKIP_TESTS=0 NO_START=0 WITH_VIEW=0 WITH_VOICE=0 POWER_REMOVE=0 SPACE_CHOICE=""
80
+ # the one web page --space lets reach bankml serve: bankML's static Hugging Face Space
81
+ SPACE_ORIGIN="https://pythai-bankml.static.hf.space"
82
  STEPS=()
83
  for a in "$@"; do
84
  case "$a" in
 
87
  --view) WITH_VIEW=1;;
88
  --voice) WITH_VOICE=1;;
89
  --remove) POWER_REMOVE=1;;
90
+ --space) SPACE_CHOICE=on;;
91
+ --no-space) SPACE_CHOICE=off;;
92
+ -h|--help) sed -n '3,21p' "$HERE/install.sh" | sed 's/^# \{0,1\}//'; exit 0;;
93
  check|build|engine|python|canon|model|voice|start|stop|status|power) STEPS+=("$a");;
94
  *) bm_die "unknown argument: $a (see ./install.sh --help)" 2;;
95
  esac
 
111
  }
112
  # shellcheck disable=SC1090
113
  [ -f "$ENV_FILE" ] && . "$ENV_FILE"
114
+ # --space / --no-space: remembered for sAGI/models.py, which starts serve; given with `start`, the running services
115
+ # stop first, so the carrier comes back with (or without) --allow-origin
116
+ if [ -n "$SPACE_CHOICE" ]; then
117
+ if [ "$SPACE_CHOICE" = on ]; then INSTALL_ALLOW_ORIGIN="$SPACE_ORIGIN"; else INSTALL_ALLOW_ORIGIN=""; fi
118
+ save_env INSTALL_ALLOW_ORIGIN "$INSTALL_ALLOW_ORIGIN"
119
+ if printf '%s\n' "${STEPS[@]}" | grep -qx start && ! printf '%s\n' "${STEPS[@]}" | grep -qx stop; then STEPS=(stop "${STEPS[@]}"); fi
120
+ fi
121
  sha256_of() { if have sha256sum; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }
122
  fetch() { # fetch URL FILE: resumable download
123
  local bar=--silent; [ -t 2 ] && bar=--progress-bar
 
296
  bm_sub "talk: $(bm_link http://127.0.0.1:7873) (this computer only)"
297
  [ "$WITH_VIEW" = 1 ] && bm_sub "watch: http://<this computer's LAN address>:7874"
298
  bm_sub "check: curl -s 127.0.0.1:18093/bankml"
299
+ if [ -n "${INSTALL_ALLOW_ORIGIN:-}" ]; then
300
+ bm_sub "web: huggingface.co/spaces/PYTHAI/bankml β†’ \"Your own bankML\" reaches this bankml serve (./install.sh start --no-space to stop it)"
301
+ fi
302
  bm_sub "docs: $(bm_link https://cryptoagi.github.io/bankml/)"
303
  }
304
 
 
328
  PY
329
  fi
330
  [ -x "$BIN" ] && bm_info "$("$BIN" version)"
331
+ bm_info "web page allowed (--space): ${INSTALL_ALLOW_ORIGIN:-none β€” this computer only}"
332
  bm_info "engine: ${LLAMA_SERVER:-not installed} Β· python: $PY"
333
  }
334
 
sAGI/models.py CHANGED
@@ -661,6 +661,15 @@ def usage() -> dict:
661
  "mem_total_gb": round(mem_total() / 1e9, 1), "mem_available_gb": round(avail / 1e9, 2), "source": "sAGI/models.py (/proc)"}
662
 
663
 
 
 
 
 
 
 
 
 
 
664
  def native_for(model: Path, engine: str | None = None) -> bool:
665
  """Whether the carrier answers from bankML's own forward pass (`bankml serve --native`, 0.3.0) for this model:
666
  "native" always (bankml refuses a model its forward pass does not run), "llama.cpp" never, "auto" for the
@@ -734,6 +743,9 @@ def _start_carrier(model: Path, fork: Path, want_sha: str | None = None, threads
734
  cmd = ([str(BANKML), "serve", str(model), "--fork", str(fork), "--spawn", str(LLAMA), "--upstream", UPSTREAM, "--listen", LISTEN,
735
  "--threads", n_threads, "--ctx", n_ctx] + (["--spec-ngram"] if resources().get("spec_ngram") else []) + ["--slot-dir", str(SLOTS)])
736
  env = None
 
 
 
737
  proc = subprocess.Popen(cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True, cwd=str(REPO), env=env)
738
  t0 = time.time()
739
  why = "timed out" # ctx: per model β€” the saved RAM budget is re-planned for this model's weights and KV size
 
661
  "mem_total_gb": round(mem_total() / 1e9, 1), "mem_available_gb": round(avail / 1e9, 2), "source": "sAGI/models.py (/proc)"}
662
 
663
 
664
+ def allow_origin() -> str | None:
665
+ """The one web page bankml serve lets in (`--allow-origin`): BANKML_ALLOW_ORIGIN, else what `./install.sh --space`
666
+ recorded; None (this computer only) when neither is set."""
667
+ o = os.environ.get("BANKML_ALLOW_ORIGIN")
668
+ if o is None:
669
+ o = _install_env(DATA).get("INSTALL_ALLOW_ORIGIN", "")
670
+ return o.strip() or None
671
+
672
+
673
  def native_for(model: Path, engine: str | None = None) -> bool:
674
  """Whether the carrier answers from bankML's own forward pass (`bankml serve --native`, 0.3.0) for this model:
675
  "native" always (bankml refuses a model its forward pass does not run), "llama.cpp" never, "auto" for the
 
743
  cmd = ([str(BANKML), "serve", str(model), "--fork", str(fork), "--spawn", str(LLAMA), "--upstream", UPSTREAM, "--listen", LISTEN,
744
  "--threads", n_threads, "--ctx", n_ctx] + (["--spec-ngram"] if resources().get("spec_ngram") else []) + ["--slot-dir", str(SLOTS)])
745
  env = None
746
+ if allow_origin():
747
+ # ./install.sh --space: bankML's Hugging Face page may talk to this serve from the browser (CORS for it alone)
748
+ cmd += ["--allow-origin", allow_origin()]
749
  proc = subprocess.Popen(cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True, cwd=str(REPO), env=env)
750
  t0 = time.time()
751
  why = "timed out" # ctx: per model β€” the saved RAM budget is re-planned for this model's weights and KV size