Your own bankML in one command: ./install.sh --space (github main a27c4b6)
Browse files- CHANGELOG.md +2 -1
- docs/install.md +1 -0
- docs/usage.md +5 -3
- index.html +6 -4
- install.sh +21 -2
- sAGI/models.py +12 -0
CHANGELOG.md
CHANGED
|
@@ -31,7 +31,8 @@ decode speed, the third, is measured on an idle machine next.
|
|
| 31 |
- `bankml serve β¦ --allow-origin ORIGIN` lets one named web page call the gateway from a browser: its CORS preflight
|
| 32 |
is answered (with Chrome's private-network grant, as a public page reaching a loopback address requires) and its
|
| 33 |
answers, streamed or not, carry `Access-Control-Allow-Origin`; any other origin gets none and a 403 preflight. The
|
| 34 |
-
loopback `Host` and JSON-POST rules are unchanged.
|
|
|
|
| 35 |
([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml)) talk to a visitor's own bankML, free, on their own
|
| 36 |
CPU β the way Savante's page reaches a local engine. Checked live: the preflight, plain and streamed answers with
|
| 37 |
the receipt, and another origin refused.
|
|
|
|
| 31 |
- `bankml serve β¦ --allow-origin ORIGIN` lets one named web page call the gateway from a browser: its CORS preflight
|
| 32 |
is answered (with Chrome's private-network grant, as a public page reaching a loopback address requires) and its
|
| 33 |
answers, streamed or not, carry `Access-Control-Allow-Origin`; any other origin gets none and a 403 preflight. The
|
| 34 |
+
loopback `Host` and JSON-POST rules are unchanged. `./install.sh start --space` turns it on for the Space (remembered
|
| 35 |
+
in `install.env`, applied to the carrier `sAGI/models.py` starts; `--no-space` turns it off). It is what lets the bankML Space's page
|
| 36 |
([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml)) talk to a visitor's own bankML, free, on their own
|
| 37 |
CPU β the way Savante's page reaches a local engine. Checked live: the preflight, plain and streamed answers with
|
| 38 |
the receipt, and another origin refused.
|
docs/install.md
CHANGED
|
@@ -135,6 +135,7 @@ alone brings the whole stack back. Use `./install.sh model start` to wait for th
|
|
| 135 |
| `--no-start` | the default run ends after `model` (or `voice`): everything installed, Savante not started. `bankml serve` is still started by `model` |
|
| 136 |
| `--view` | `start` also starts view mode on `0.0.0.0:7874`. It works with named steps too: `./install.sh start --view` |
|
| 137 |
| `--voice` | the default run includes `voice`. With named steps, name it instead: `./install.sh voice` |
|
|
|
|
| 138 |
| `-h`, `--help` | prints the header of `install.sh` and exits |
|
| 139 |
|
| 140 |
Any other argument stops the installer with exit code 2.
|
|
|
|
| 135 |
| `--no-start` | the default run ends after `model` (or `voice`): everything installed, Savante not started. `bankml serve` is still started by `model` |
|
| 136 |
| `--view` | `start` also starts view mode on `0.0.0.0:7874`. It works with named steps too: `./install.sh start --view` |
|
| 137 |
| `--voice` | the default run includes `voice`. With named steps, name it instead: `./install.sh voice` |
|
| 138 |
+
| `--space` / `--no-space` | 0.3.9: let bankML's Hugging Face page ([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml)) talk to your `bankml serve` from your browser β its *Your own bankML* mode: your CPU, your verified model, a receipt the page checks. Serve starts with `--allow-origin https://pythai-bankml.static.hf.space` (that one page, nothing else); the choice is remembered in `install.env` (`INSTALL_ALLOW_ORIGIN`; `BANKML_ALLOW_ORIGIN` overrides it). `./install.sh start --space` applies it at once (the running services stop first); `--no-space` forgets it. `./install.sh status` says which page is allowed |
|
| 139 |
| `-h`, `--help` | prints the header of `install.sh` and exits |
|
| 140 |
|
| 141 |
Any other argument stops the installer with exit code 2.
|
docs/usage.md
CHANGED
|
@@ -247,9 +247,11 @@ In the next release (not yet tagged); each is checked against llama-server b1119
|
|
| 247 |
llama-server `-np 1`. Like llama-server, it keeps the states of other conversations in RAM (`BANKML_CACHE_RAM`, MiB;
|
| 248 |
0 off, -1 no limit; unset, 8192 MiB but at most a quarter of the memory available at load), so a conversation that comes back after
|
| 249 |
another does not recompute its whole history.
|
| 250 |
-
- **Your bankML from a web page (0.3.9).** `
|
| 251 |
-
|
| 252 |
-
|
|
|
|
|
|
|
| 253 |
- **A smaller conversation memory (0.3.9, in progress).** `BANKML_CACHE_TYPE=q8_0` keeps the KV cache in q8_0 instead of f16,
|
| 254 |
about half the memory, so a long context fits on a small machine. It is llama.cpp's `--cache-type-k q8_0
|
| 255 |
--cache-type-v q8_0` exactly, with the Hadamard rotation llama.cpp applies around a quantized cache, and gives the
|
|
|
|
| 247 |
llama-server `-np 1`. Like llama-server, it keeps the states of other conversations in RAM (`BANKML_CACHE_RAM`, MiB;
|
| 248 |
0 off, -1 no limit; unset, 8192 MiB but at most a quarter of the memory available at load), so a conversation that comes back after
|
| 249 |
another does not recompute its whole history.
|
| 250 |
+
- **Your bankML from a web page (0.3.9).** `./install.sh start --space` lets the bankML Space's page
|
| 251 |
+
([PYTHAI/bankml](https://huggingface.co/spaces/PYTHAI/bankml), *Your own bankML*) talk to your own `bankml serve`
|
| 252 |
+
from your browser: your CPU, your verified model, a receipt on every answer, nothing sent anywhere else. It starts
|
| 253 |
+
serve with `--allow-origin https://pythai-bankml.static.hf.space`, so only that one page gets CORS headers;
|
| 254 |
+
`./install.sh start --no-space` takes it back.
|
| 255 |
- **A smaller conversation memory (0.3.9, in progress).** `BANKML_CACHE_TYPE=q8_0` keeps the KV cache in q8_0 instead of f16,
|
| 256 |
about half the memory, so a long context fits on a small machine. It is llama.cpp's `--cache-type-k q8_0
|
| 257 |
--cache-type-v q8_0` exactly, with the Hadamard rotation llama.cpp applies around a quantized cache, and gives the
|
index.html
CHANGED
|
@@ -184,10 +184,12 @@
|
|
| 184 |
<details>
|
| 185 |
<summary>Start your own bankML for this page (Linux, AVX2; free)</summary>
|
| 186 |
<ol>
|
| 187 |
-
<li><code>git clone https://github.com/cryptoAGI/bankml && cd bankml && ./install.sh</code> β builds bankML
|
| 188 |
-
|
| 189 |
-
<code>
|
| 190 |
-
<li>Press <b>connect</b>. Your browser may ask to let this page reach your
|
|
|
|
|
|
|
| 191 |
</ol>
|
| 192 |
</details>
|
| 193 |
</div>
|
|
|
|
| 184 |
<details>
|
| 185 |
<summary>Start your own bankML for this page (Linux, AVX2; free)</summary>
|
| 186 |
<ol>
|
| 187 |
+
<li><code>git clone https://github.com/cryptoAGI/bankml && cd bankml && ./install.sh --space</code> β builds bankML,
|
| 188 |
+
imports and verifies Bonsai-8B, and starts it so this page (<span id="origin"></span>) may talk to it. Already installed?
|
| 189 |
+
<code>git pull && ./install.sh build start --space</code></li>
|
| 190 |
+
<li>Press <b>connect</b>. Your browser may ask to let this page reach devices on your local network: that is the
|
| 191 |
+
request for your own bankML on 127.0.0.1, and nothing else.</li>
|
| 192 |
+
<li><code>./install.sh start --no-space</code> takes the permission back.</li>
|
| 193 |
</ol>
|
| 194 |
</details>
|
| 195 |
</div>
|
install.sh
CHANGED
|
@@ -11,6 +11,10 @@
|
|
| 11 |
# --no-start install everything, start nothing
|
| 12 |
# --view also start view mode for the LAN (0.0.0.0:7874)
|
| 13 |
# --voice also install Savante's Piper voice (step `voice`)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
# power [--remove] opt-in, sudo: let your user read the CPU package energy counter (RAPL), so bankML can measure
|
| 15 |
# watts and joules per token; it asks first (PLATYPUS, CVE-2020-8694), and --remove undoes it
|
| 16 |
# -h, --help this text
|
|
@@ -72,7 +76,9 @@ else CANON="$HOME/cryptoAGI/savante"; fi
|
|
| 72 |
LOGS="$DATA/logs"
|
| 73 |
|
| 74 |
# ββ options ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 75 |
-
SKIP_TESTS=0 NO_START=0 WITH_VIEW=0 WITH_VOICE=0 POWER_REMOVE=0
|
|
|
|
|
|
|
| 76 |
STEPS=()
|
| 77 |
for a in "$@"; do
|
| 78 |
case "$a" in
|
|
@@ -81,7 +87,9 @@ for a in "$@"; do
|
|
| 81 |
--view) WITH_VIEW=1;;
|
| 82 |
--voice) WITH_VOICE=1;;
|
| 83 |
--remove) POWER_REMOVE=1;;
|
| 84 |
-
-
|
|
|
|
|
|
|
| 85 |
check|build|engine|python|canon|model|voice|start|stop|status|power) STEPS+=("$a");;
|
| 86 |
*) bm_die "unknown argument: $a (see ./install.sh --help)" 2;;
|
| 87 |
esac
|
|
@@ -103,6 +111,13 @@ save_env() { # save_env KEY VALUE: remember a choice in $ENV_FILE
|
|
| 103 |
}
|
| 104 |
# shellcheck disable=SC1090
|
| 105 |
[ -f "$ENV_FILE" ] && . "$ENV_FILE"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
sha256_of() { if have sha256sum; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }
|
| 107 |
fetch() { # fetch URL FILE: resumable download
|
| 108 |
local bar=--silent; [ -t 2 ] && bar=--progress-bar
|
|
@@ -281,6 +296,9 @@ step_start() {
|
|
| 281 |
bm_sub "talk: $(bm_link http://127.0.0.1:7873) (this computer only)"
|
| 282 |
[ "$WITH_VIEW" = 1 ] && bm_sub "watch: http://<this computer's LAN address>:7874"
|
| 283 |
bm_sub "check: curl -s 127.0.0.1:18093/bankml"
|
|
|
|
|
|
|
|
|
|
| 284 |
bm_sub "docs: $(bm_link https://cryptoagi.github.io/bankml/)"
|
| 285 |
}
|
| 286 |
|
|
@@ -310,6 +328,7 @@ print(" verified:", s.get("verified"), "Β·", s.get("model", "?").rsplit("/", 1
|
|
| 310 |
PY
|
| 311 |
fi
|
| 312 |
[ -x "$BIN" ] && bm_info "$("$BIN" version)"
|
|
|
|
| 313 |
bm_info "engine: ${LLAMA_SERVER:-not installed} Β· python: $PY"
|
| 314 |
}
|
| 315 |
|
|
|
|
| 11 |
# --no-start install everything, start nothing
|
| 12 |
# --view also start view mode for the LAN (0.0.0.0:7874)
|
| 13 |
# --voice also install Savante's Piper voice (step `voice`)
|
| 14 |
+
# --space let bankML's Hugging Face page (huggingface.co/spaces/PYTHAI/bankml) talk to your bankml serve from
|
| 15 |
+
# your browser ("Your own bankML": your CPU, your verified model, a receipt): serve starts with
|
| 16 |
+
# --allow-origin for that one page; remembered; `./install.sh start --space` applies it now
|
| 17 |
+
# --no-space forget it (serve answers this computer only, no web page)
|
| 18 |
# power [--remove] opt-in, sudo: let your user read the CPU package energy counter (RAPL), so bankML can measure
|
| 19 |
# watts and joules per token; it asks first (PLATYPUS, CVE-2020-8694), and --remove undoes it
|
| 20 |
# -h, --help this text
|
|
|
|
| 76 |
LOGS="$DATA/logs"
|
| 77 |
|
| 78 |
# ββ options ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 79 |
+
SKIP_TESTS=0 NO_START=0 WITH_VIEW=0 WITH_VOICE=0 POWER_REMOVE=0 SPACE_CHOICE=""
|
| 80 |
+
# the one web page --space lets reach bankml serve: bankML's static Hugging Face Space
|
| 81 |
+
SPACE_ORIGIN="https://pythai-bankml.static.hf.space"
|
| 82 |
STEPS=()
|
| 83 |
for a in "$@"; do
|
| 84 |
case "$a" in
|
|
|
|
| 87 |
--view) WITH_VIEW=1;;
|
| 88 |
--voice) WITH_VOICE=1;;
|
| 89 |
--remove) POWER_REMOVE=1;;
|
| 90 |
+
--space) SPACE_CHOICE=on;;
|
| 91 |
+
--no-space) SPACE_CHOICE=off;;
|
| 92 |
+
-h|--help) sed -n '3,21p' "$HERE/install.sh" | sed 's/^# \{0,1\}//'; exit 0;;
|
| 93 |
check|build|engine|python|canon|model|voice|start|stop|status|power) STEPS+=("$a");;
|
| 94 |
*) bm_die "unknown argument: $a (see ./install.sh --help)" 2;;
|
| 95 |
esac
|
|
|
|
| 111 |
}
|
| 112 |
# shellcheck disable=SC1090
|
| 113 |
[ -f "$ENV_FILE" ] && . "$ENV_FILE"
|
| 114 |
+
# --space / --no-space: remembered for sAGI/models.py, which starts serve; given with `start`, the running services
|
| 115 |
+
# stop first, so the carrier comes back with (or without) --allow-origin
|
| 116 |
+
if [ -n "$SPACE_CHOICE" ]; then
|
| 117 |
+
if [ "$SPACE_CHOICE" = on ]; then INSTALL_ALLOW_ORIGIN="$SPACE_ORIGIN"; else INSTALL_ALLOW_ORIGIN=""; fi
|
| 118 |
+
save_env INSTALL_ALLOW_ORIGIN "$INSTALL_ALLOW_ORIGIN"
|
| 119 |
+
if printf '%s\n' "${STEPS[@]}" | grep -qx start && ! printf '%s\n' "${STEPS[@]}" | grep -qx stop; then STEPS=(stop "${STEPS[@]}"); fi
|
| 120 |
+
fi
|
| 121 |
sha256_of() { if have sha256sum; then sha256sum "$1" | cut -d' ' -f1; else shasum -a 256 "$1" | cut -d' ' -f1; fi; }
|
| 122 |
fetch() { # fetch URL FILE: resumable download
|
| 123 |
local bar=--silent; [ -t 2 ] && bar=--progress-bar
|
|
|
|
| 296 |
bm_sub "talk: $(bm_link http://127.0.0.1:7873) (this computer only)"
|
| 297 |
[ "$WITH_VIEW" = 1 ] && bm_sub "watch: http://<this computer's LAN address>:7874"
|
| 298 |
bm_sub "check: curl -s 127.0.0.1:18093/bankml"
|
| 299 |
+
if [ -n "${INSTALL_ALLOW_ORIGIN:-}" ]; then
|
| 300 |
+
bm_sub "web: huggingface.co/spaces/PYTHAI/bankml β \"Your own bankML\" reaches this bankml serve (./install.sh start --no-space to stop it)"
|
| 301 |
+
fi
|
| 302 |
bm_sub "docs: $(bm_link https://cryptoagi.github.io/bankml/)"
|
| 303 |
}
|
| 304 |
|
|
|
|
| 328 |
PY
|
| 329 |
fi
|
| 330 |
[ -x "$BIN" ] && bm_info "$("$BIN" version)"
|
| 331 |
+
bm_info "web page allowed (--space): ${INSTALL_ALLOW_ORIGIN:-none β this computer only}"
|
| 332 |
bm_info "engine: ${LLAMA_SERVER:-not installed} Β· python: $PY"
|
| 333 |
}
|
| 334 |
|
sAGI/models.py
CHANGED
|
@@ -661,6 +661,15 @@ def usage() -> dict:
|
|
| 661 |
"mem_total_gb": round(mem_total() / 1e9, 1), "mem_available_gb": round(avail / 1e9, 2), "source": "sAGI/models.py (/proc)"}
|
| 662 |
|
| 663 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 664 |
def native_for(model: Path, engine: str | None = None) -> bool:
|
| 665 |
"""Whether the carrier answers from bankML's own forward pass (`bankml serve --native`, 0.3.0) for this model:
|
| 666 |
"native" always (bankml refuses a model its forward pass does not run), "llama.cpp" never, "auto" for the
|
|
@@ -734,6 +743,9 @@ def _start_carrier(model: Path, fork: Path, want_sha: str | None = None, threads
|
|
| 734 |
cmd = ([str(BANKML), "serve", str(model), "--fork", str(fork), "--spawn", str(LLAMA), "--upstream", UPSTREAM, "--listen", LISTEN,
|
| 735 |
"--threads", n_threads, "--ctx", n_ctx] + (["--spec-ngram"] if resources().get("spec_ngram") else []) + ["--slot-dir", str(SLOTS)])
|
| 736 |
env = None
|
|
|
|
|
|
|
|
|
|
| 737 |
proc = subprocess.Popen(cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True, cwd=str(REPO), env=env)
|
| 738 |
t0 = time.time()
|
| 739 |
why = "timed out" # ctx: per model β the saved RAM budget is re-planned for this model's weights and KV size
|
|
|
|
| 661 |
"mem_total_gb": round(mem_total() / 1e9, 1), "mem_available_gb": round(avail / 1e9, 2), "source": "sAGI/models.py (/proc)"}
|
| 662 |
|
| 663 |
|
| 664 |
+
def allow_origin() -> str | None:
|
| 665 |
+
"""The one web page bankml serve lets in (`--allow-origin`): BANKML_ALLOW_ORIGIN, else what `./install.sh --space`
|
| 666 |
+
recorded; None (this computer only) when neither is set."""
|
| 667 |
+
o = os.environ.get("BANKML_ALLOW_ORIGIN")
|
| 668 |
+
if o is None:
|
| 669 |
+
o = _install_env(DATA).get("INSTALL_ALLOW_ORIGIN", "")
|
| 670 |
+
return o.strip() or None
|
| 671 |
+
|
| 672 |
+
|
| 673 |
def native_for(model: Path, engine: str | None = None) -> bool:
|
| 674 |
"""Whether the carrier answers from bankML's own forward pass (`bankml serve --native`, 0.3.0) for this model:
|
| 675 |
"native" always (bankml refuses a model its forward pass does not run), "llama.cpp" never, "auto" for the
|
|
|
|
| 743 |
cmd = ([str(BANKML), "serve", str(model), "--fork", str(fork), "--spawn", str(LLAMA), "--upstream", UPSTREAM, "--listen", LISTEN,
|
| 744 |
"--threads", n_threads, "--ctx", n_ctx] + (["--spec-ngram"] if resources().get("spec_ngram") else []) + ["--slot-dir", str(SLOTS)])
|
| 745 |
env = None
|
| 746 |
+
if allow_origin():
|
| 747 |
+
# ./install.sh --space: bankML's Hugging Face page may talk to this serve from the browser (CORS for it alone)
|
| 748 |
+
cmd += ["--allow-origin", allow_origin()]
|
| 749 |
proc = subprocess.Popen(cmd, stdout=log, stderr=log, stdin=subprocess.DEVNULL, start_new_session=True, cwd=str(REPO), env=env)
|
| 750 |
t0 = time.time()
|
| 751 |
why = "timed out" # ctx: per model β the saved RAM budget is re-planned for this model's weights and KV size
|