v1.2: TypeSafe API wire compatibility in serve.py (server-side SDKs, GET /v1/models, optional API key, error detail, request ids; a full batching queue answers 529); score answers carry confidence; weights unchanged

README.md

@@ -203,8 +203,8 @@ import os, sys
 from huggingface_hub import hf_hub_download
 
 os.environ["BLINK_MODEL"] = "thegovind/blink-mimo-9b"
-os.environ["BLINK_REVISION"] = "v1.0"
-sys.path.insert(0, os.path.dirname(hf_hub_download("thegovind/blink-mimo-9b", "blink.py", revision="v1.0")))
+os.environ["BLINK_REVISION"] = "v1.2"
+sys.path.insert(0, os.path.dirname(hf_hub_download("thegovind/blink-mimo-9b", "blink.py", revision="v1.2")))
 import blink
 
 out = blink.decide(
@@ -224,14 +224,17 @@ print(out["answers"]["intent"]["probabilities"])
 
 ## Run it as a server
 
-`serve.py` accepts Jev-compatible `POST /v1/systemone` (`{state, questions}` → `{answers, usage}`);
-JevBench's stock `typesafe` adapter and the Decision Index kit's `http` engine use this format.
-`GET /healthz` reports startup checks.
+`serve.py` handles TypeSafe's request and answer fields at `POST /v1/systemone` and lists its model at
+`GET /v1/models`. From server-side code, point TypeSafe's Python or JavaScript SDK at the server with
+`TYPESAFE_BASE_URL`;
+JevBench's stock `typesafe` adapter and the Decision Index kit's `http` engine still work unchanged.
+`GET /healthz` reports startup checks. `v1.2` changes code only; its weights are identical to `v1.0`.
 
 ```sh
 pip install "torch==2.13.0" "transformers==5.17.0" "flash-linear-attention==0.5.2" "accelerate>=1.1.0" safetensors huggingface_hub
-hf download thegovind/blink-mimo-9b --revision v1.0 --local-dir blink-mimo-9b
+hf download thegovind/blink-mimo-9b --revision v1.2 --local-dir blink-mimo-9b
 python blink-mimo-9b/serve.py --model ./blink-mimo-9b --port 8000
+# TypeSafe SDKs: export TYPESAFE_BASE_URL=http://127.0.0.1:8000 TYPESAFE_API_KEY=any
 ```
 
 In another terminal: `curl -s http://127.0.0.1:8000/healthz`.
@@ -250,25 +253,30 @@ docker build -t blink-mimo-9b . && docker run --rm --gpus all -p 127.0.0.1:8000:
   answers), `kernels` (fast path or slower fallback without flash-linear-attention), `versions` and `hub_offline`.
 - Limits: 255 options per choice, 2–10 score levels, 131,072 input tokens per question and 512 questions
   per request. Over-limit requests get HTTP 422 with the reason; nothing is truncated.
+- `GET /v1/models` lists the one served model with a blank `release_date`. Every request uses that model
+  regardless of its `model` field.
+- The server is open by default. Set `--api-key` or `BLINK_API_KEY` to require `Authorization: Bearer <key>` on both API
+  routes. Missing or wrong keys get 401; `/healthz` stays open.
+- Error bodies put the reason in `error` and `detail`. Over-limit requests return 422, with nothing cut.
 - Requests run one at a time. Questions are batched; each batch takes one forward pass (large requests
   can take more than one). Serving the downloaded folder or Docker image enables Hugging Face offline
   mode before model loading (`hub_offline: true`). The server doesn't otherwise restrict network access.
 - Set `--batch-window-ms 5` to turn on cross-request batching with a 5 ms collection window, up to
   `--max-batch-requests` requests at a time, which defaults to 16. The window defaults to 0, so requests still
   run one at a time. `--max-queued-requests` lets up to 64 requests wait for a batch by default. Excess requests
-  get HTTP 503 with `Retry-After`, so clients should retry. On a 1,000-request Decision Index sample over HTTP,
-  throughput rose about 15% with 4 concurrent clients and 16% with 16. One client saw no gain. Offline runs on
-  long documents showed no meaningful gain. On the Decision Index sample, a set of long workflow documents, and
-  the public TypeSafe cases covered by the FP32 reference, batched answers passed the same numerical-parity
-  checks against an FP32 reference as one-at-a-time answers, covering argmax agreement and probability
-  differences. TypeSafe documents were sent as JSON objects. The FP32 reference could not run the five longest
-  TypeSafe documents, so parity covered 170 of 354 TypeSafe questions. Batched and one-at-a-time answers chose
-  the same option every time on the other 184. A few near-tied answers can still flip. `v1.1` changes code only,
-  leaving `v1.0` weights unchanged. Update the two code files in an existing `v1.0` download, then restart with
-  the flag:
+  get HTTP 529 with `Retry-After`, so clients should retry. `v1.1` returned HTTP 503 for a full queue. On a
+  1,000-request Decision Index sample over HTTP, throughput rose about 15% with 4 concurrent clients and 16%
+  with 16. One client saw no gain. Offline runs on long documents showed no meaningful gain. On the Decision
+  Index sample, a set of long workflow documents, and the public TypeSafe cases covered by the FP32 reference,
+  batched answers passed the same numerical-parity checks against an FP32 reference as one-at-a-time answers,
+  covering argmax agreement and probability differences. TypeSafe documents were sent as JSON objects. The FP32
+  reference could not run the five longest TypeSafe documents, so parity covered 170 of 354 TypeSafe questions.
+  Batched and one-at-a-time answers chose the same option every time on the other 184. A few near-tied answers
+  can still flip. Batching arrived in `v1.1`. The current code revision is `v1.2`, with weights identical to
+  `v1.0`. Update the two code files in an existing `v1.0` download, then restart with the flag:
 
   ```sh
-  hf download thegovind/blink-mimo-9b serve.py blink.py --revision v1.1 --local-dir blink-mimo-9b
+  hf download thegovind/blink-mimo-9b serve.py blink.py --revision v1.2 --local-dir blink-mimo-9b
   python blink-mimo-9b/serve.py --model ./blink-mimo-9b --port 8000 --batch-window-ms 5
   ```
 - blink-mimo-9b weights are 18.8 GB in bf16. Long prompts need more memory.

blink.py

@@ -223,12 +223,16 @@ def answer_for(q: dict, keys: list[str], probs: list[float]) -> dict:
         return {"type": "noul", "noul": p["yes"], "probabilities": p}
     levels = [str(i) for i in range(len(q["criteria"]))]
     ev = sum(int(k) * p[k] for k in levels)
+    top = max(levels, key=lambda k: (p[k], -int(k)))
+    K = len(levels)
     return {
         "type": "score",
         "score": ev,
         "probabilities": {k: p[k] for k in levels},
         "legend": {k: text(q["criteria"][int(k)]) for k in levels},
-        "choice": max(levels, key=lambda k: (p[k], -int(k))),
+        "choice": top,
+        # added after the fields above, which are unchanged: the choice formula over the levels (TypeSafe-shaped)
+        "confidence": min(1.0, max(0.0, (p[top] - 1 / K) / (1 - 1 / K))),
     }
 
 

serve.py

@@ -1,32 +1,44 @@
-"""blink server: a Jev-compatible decision endpoint.
+"""blink server: a Jev-compatible decision endpoint (the TypeSafe API's wire format).
 
     pip install "torch==2.13.0" "transformers==5.17.0" "flash-linear-attention==0.5.2" accelerate safetensors huggingface_hub
     hf download thegovind/blink-4b --revision v1.0 --local-dir blink-4b
     python blink-4b/serve.py --model ./blink-4b --port 8000
 
-POST /v1/systemone  {"state": ..., "questions": {...}}  ->  {"model", "answers", "usage"}
-GET  /healthz       ->  {"ok", "model", "revision", "weights_verified", "hub_offline", "warmup", "kernels", "versions"}
-
-Requests are served one at a time. A request over a limit (options per choice, context length,
-questions per request) gets HTTP 422 with the reason; nothing is truncated. With weights.sha256 beside
-the weights, every listed file is hashed before serving (weights_verified). Serving a local folder switches
-the Hugging Face libraries to offline mode before any of them loads (hub_offline reports the setting the
-libraries actually use); the server does not otherwise restrict the network.
+POST /v1/systemone  {"state": ..., "model": ..., "questions": {...}}  ->  {"model", "answers", "usage"}
+GET  /v1/models     ->  {"models": [{"name", "description", "release_date"}]}
+GET  /healthz       ->  {"ok", "model", "revision", "weights_verified", "hub_offline", "warmup", "kernels", "versions",
+                         "batching", "api_key_required"}
+
+A client written for the TypeSafe API works unchanged against this server once its base URL points here (for the
+official SDKs: TYPESAFE_BASE_URL=http://127.0.0.1:8000). The request's "model" is accepted and not used: this
+server answers with the one model it serves and names it in the response. Any Authorization header is accepted and
+ignored unless a key is set with --api-key or BLINK_API_KEY; then /v1/systemone and /v1/models need
+"Authorization: Bearer <key>" and answer HTTP 401 without it (/healthz stays open).
+
+Requests are served one at a time. A request blink can't answer (a malformed question, or one over a limit: options
+per choice, context length, questions per request) gets HTTP 422 with the reason; nothing is truncated. A body that
+isn't a JSON object gets HTTP 400. Error bodies are {"error": reason, "detail": ...}, where detail is a list of
+{"loc", "msg", "type"} for a 400 or 422 and the reason again otherwise. Every response carries an
+x-typesafe-request-id header. With weights.sha256 beside the weights, every listed file is hashed before serving
+(weights_verified). Serving a local folder switches the Hugging Face libraries to offline mode before any of them
+loads (hub_offline reports the setting the libraries actually use); the server does not otherwise restrict the network.
 
 Opt-in cross-request batching (--batch-window-ms, default 0 = off): requests that arrive within the window are
 decided together, up to --max-batch-requests; at most --max-queued-requests wait, and past that a request gets
-HTTP 503 with Retry-After. Each request still gets its own answers or its own error.
+HTTP 529 (overloaded) with Retry-After. Each request still gets its own answers or its own error.
 """
 
 from __future__ import annotations
 
 import argparse
 import hashlib
+import hmac
 import json
 import os
 import socket
 import sys
 import threading
+import uuid
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
 WARMUP = ("Order 4471 arrived with a cracked screen. The customer attached photos and wants a replacement.",
@@ -95,6 +107,42 @@ def int_range(lo: int, hi: int):
     return parse
 
 
+def api_key(value: str):
+    """--api-key / BLINK_API_KEY: empty means open; a key must fit in an Authorization header (printable ASCII, no
+    whitespace), as TypeSafe's SDKs require of theirs."""
+    key = value.strip()
+    if not key:
+        return None
+    if not (key.isascii() and key.isprintable()) or " " in key:
+        raise argparse.ArgumentTypeError("must be printable ASCII without whitespace")
+    return key
+
+
+def bearer(header) -> str | None:
+    """The token of an "Authorization: Bearer <token>" header, else None."""
+    scheme, _, token = (header or "").strip().partition(" ")
+    if scheme.lower() != "bearer":
+        return None
+    return token.strip() or None
+
+
+def error_loc(blink, req: dict, message: str) -> list:
+    """Where a refused request went wrong, as a FastAPI-style location (TypeSafe's 422 body): the question the reason
+    opens with (blink's refusals that name one start "question '<key>' "), else the first question blink can't read,
+    else the questions map. Only the opening counts: a key quoted later in a reason, say inside a bad type, isn't it."""
+    questions = req.get("questions")
+    if isinstance(questions, dict) and 0 < len(questions) <= getattr(blink, "MAX_QUESTIONS", 512):
+        for key in questions:
+            if message.startswith(f"question {key!r} "):
+                return ["body", "questions", key]
+        for key, q in questions.items():
+            try:
+                blink.question_options(q)
+            except Exception:  # noqa: BLE001 - only locating the refusal, which is already decided
+                return ["body", "questions", key]
+    return ["body", "questions"]
+
+
 def main() -> None:
     ap = argparse.ArgumentParser(description="Serve blink over a Jev-compatible HTTP API.")
     ap.add_argument("--model", default=os.environ.get("BLINK_MODEL", "thegovind/blink-4b"))
@@ -108,7 +156,11 @@ def main() -> None:
     ap.add_argument("--max-batch-requests", type=int_range(1, 64), default=16,
                     help="most requests decided together (1-64)")
     ap.add_argument("--max-queued-requests", type=int_range(1, 1024), default=64,
-                    help="most requests waiting for a batch; past this a request gets HTTP 503 (1-1024)")
+                    help="most requests waiting for a batch; past this a request gets HTTP 529 (1-1024)")
+    # a string default goes through api_key() too, so BLINK_API_KEY is checked the same way
+    ap.add_argument("--api-key", type=api_key, default=os.environ.get("BLINK_API_KEY", ""),
+                    help="require 'Authorization: Bearer <key>' on /v1/systemone and /v1/models (default: open, any "
+                         "Authorization header is accepted and ignored); BLINK_API_KEY keeps it out of the process list")
     a = ap.parse_args()
 
     local = os.path.isdir(a.model)
@@ -151,6 +203,14 @@ def main() -> None:
                if a.batch_window_ms > 0 else None)
     health["batching"] = ({"window_ms": a.batch_window_ms, "max_requests": a.max_batch_requests,
                            "max_queued": a.max_queued_requests} if batcher else None)
+    health["api_key_required"] = a.api_key is not None
+    # TypeSafe's model list, for clients that ask which names the model field takes; any name is accepted here
+    listing = {"models": [{
+        "name": a.model,
+        "description": "blink: typed decisions (noul, choice, score) with option probabilities from one forward pass. "
+                       "This server serves one model; a request's model field is accepted and not used.",
+        "release_date": "",
+    }]}
 
     def current_health() -> dict:
         if batcher is None:
@@ -171,31 +231,61 @@ def main() -> None:
             # on the client's delayed ACK, a flat ~40 ms on every request of a kept-alive connection
             self.connection.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
 
-        def _send(self, code: int, obj: dict, headers: dict | None = None) -> None:
+        def _send(self, code: int, obj: dict, headers: dict | None = None, reason: str | None = None) -> None:
             body = json.dumps(obj, ensure_ascii=False).encode("utf-8")
-            self.send_response(code)
+            self.send_response(code, reason)
             self.send_header("Content-Type", "application/json")
             self.send_header("Content-Length", str(len(body)))
+            self.send_header("x-typesafe-request-id", uuid.uuid4().hex)  # read by TypeSafe's SDKs
             for name, value in (headers or {}).items():
                 self.send_header(name, value)
             self.end_headers()
             self.wfile.write(body)
 
+        def _fail(self, code: int, reason: str, detail=None, headers: dict | None = None, *, unread: bool = False,
+                  status_text: str | None = None) -> None:
+            """An error body: blink's {"error"} plus TypeSafe's (FastAPI's) {"detail"}. With the request body still
+            unread, the connection is closed rather than left holding it."""
+            if unread:
+                self.close_connection = True
+                headers = {**(headers or {}), "Connection": "close"}
+            self._send(code, {"error": reason, "detail": reason if detail is None else detail}, headers, status_text)
+
+        def _refused(self, code: int, reason: str, loc: list, kind: str) -> None:
+            self._fail(code, reason, [{"loc": loc, "msg": reason, "type": kind}])
+
+        def _authorized(self) -> bool:
+            if a.api_key is None:
+                return True
+            token = bearer(self.headers.get("Authorization"))
+            return token is not None and hmac.compare_digest(token.encode("utf-8"), a.api_key.encode("utf-8"))
+
+        def _unauthorized(self, unread: bool) -> None:
+            self._fail(401, "missing or invalid API key: send Authorization: Bearer <key>",
+                       headers={"WWW-Authenticate": "Bearer"}, unread=unread)
+
         def do_GET(self):
-            if self.path.rstrip("/") in ("/healthz", "/health"):
+            path = self.path.rstrip("/")
+            if path in ("/healthz", "/health"):
                 return self._send(200, current_health())
-            return self._send(404, {"error": "not found"})
+            if path == "/v1/models":
+                if not self._authorized():
+                    return self._unauthorized(unread=False)
+                return self._send(200, listing)
+            return self._fail(404, "not found")
 
         def do_POST(self):
             if self.path.rstrip("/") != "/v1/systemone":
-                return self._send(404, {"error": "not found"})
+                return self._fail(404, "not found", unread=True)
+            if not self._authorized():
+                return self._unauthorized(unread=True)
             try:
                 size = int(self.headers.get("Content-Length") or 0)
                 req = json.loads(self.rfile.read(size) or b"{}")
             except (ValueError, json.JSONDecodeError) as exc:
-                return self._send(400, {"error": f"invalid JSON: {exc}"})
+                return self._refused(400, f"invalid JSON: {exc}", ["body"], "json_invalid")
             if not isinstance(req, dict):
-                return self._send(400, {"error": "the body must be a JSON object"})
+                return self._refused(400, "the body must be a JSON object", ["body"], "value_error")
             try:
                 if batcher is not None:
                     out = batcher.submit(req.get("state"), req.get("questions"))
@@ -203,11 +293,11 @@ def main() -> None:
                     with lock:
                         out = blink.decide(req.get("state"), req.get("questions"))
             except blink.BlinkError as exc:
-                return self._send(422, {"error": str(exc)})
+                return self._refused(422, str(exc), error_loc(blink, req, str(exc)), "value_error")
             except busy as exc:
-                return self._send(503, {"error": str(exc)}, {"Retry-After": "1"})
+                return self._fail(529, str(exc), headers={"Retry-After": "1"}, status_text="Overloaded")
             except Exception as exc:  # noqa: BLE001 - report, keep serving
-                return self._send(500, {"error": f"{type(exc).__name__}: {exc}"})
+                return self._fail(500, f"{type(exc).__name__}: {exc}")
             return self._send(200, {
                 "model": a.model,
                 "answers": out["answers"],
@@ -215,7 +305,8 @@ def main() -> None:
             })
 
     server = ThreadingHTTPServer((a.host, a.port), Handler)
-    print(f"blink serving {a.model} on http://{a.host}:{a.port} ({kernels}; weights_verified={verified})", flush=True)
+    print(f"blink serving {a.model} on http://{a.host}:{a.port} ({kernels}; weights_verified={verified}; "
+          f"api key {'required' if a.api_key else 'not required'})", flush=True)
     server.serve_forever()