Sync to v0.2.2 (github.com/iscc/iscc-sct @ bf39b3a)

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+docs/assets/logo_dark.png filter=lfs diff=lfs merge=lfs -text
+docs/assets/logo_light.png filter=lfs diff=lfs merge=lfs -text

.pre-commit-config.yaml

@@ -0,0 +1,36 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0
+    hooks:
+      - id: trailing-whitespace
+        exclude: ^(iscc_sct/tokenizer\.json|tests/.*\.txt)$
+      - id: check-added-large-files
+        exclude: ^(uv\.lock|iscc_sct/tokenizer\.json)$
+      - id: end-of-file-fixer
+        exclude: ^(iscc_sct/tokenizer\.json|tests/.*\.txt)$
+      - id: fix-byte-order-marker
+        exclude: ^tests/.*\.txt$
+      - id: check-json
+      - id: check-toml
+      - id: check-yaml
+      - id: mixed-line-ending
+        args: [--fix=lf]
+        exclude: ^(iscc_sct/tokenizer\.json|tests/.*\.txt)$
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.16
+    hooks:
+      - id: ruff-format
+      - id: ruff-check
+        args: [--fix]
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 1.0.0
+    hooks:
+      - id: mdformat
+        args: [--wrap=100, --end-of-line=lf]
+        # api.md uses mkdocstrings ::: directives that mdformat would collapse and break.
+        exclude: ^docs/reference/api\.md$
+        additional_dependencies:
+          # mdformat-mkdocs[recommended] bundles mdformat-gfm and frontmatter support and adds
+          # mkdocs admonition handling (!!!/???); gfm-alerts keeps the README's GitHub alerts
+          - "mdformat-mkdocs[recommended]"
+          - mdformat-gfm-alerts

CHANGELOG.md

@@ -1,17 +1,124 @@
 # Changelog
 
-## [0.1.3] - Unrelease
+## [0.2.2] - 2026-08-03
+
+- Changed the default inference batch size from a fixed 100 to auto (`batch_size=0`), which resolves
+    to 1 chunk per batch on CPU and 100 on CUDA (#25). The vendored tokenizer pads each batch to its
+    longest chunk and attention cost is quadratic in sequence length, so on CPU large batches mostly
+    pay for padding. Measured on a 2238-chunk book: ~1.4x faster and 1529 MB → 1110 MB steady RSS
+    single-process; on a 12-worker pool 1.45x throughput and 18.5 GB → 13.4 GB total PSS. Generated
+    ISCC codes are unchanged (verified bit-identical at 256 bits across ~50 configurations, and
+    identical between the CPU and CUDA providers). A GPU scales the opposite way, which is why the
+    default is provider-aware rather than one global value: on a GTX 1080, throughput rises from 384
+    chunks/s at batch 1 to 668 at batch 100 and then plateaus (batch 200 is no faster and nearly
+    doubles VRAM), so the CUDA path keeps batch 100. Host RSS on the GPU path is unaffected by batch
+    size — those activations live in VRAM
+- Added the `batch_size` option (`ISCC_SCT_BATCH_SIZE`) to override the batch size per call or
+    globally, and `intra_op_threads` (`ISCC_SCT_INTRA_OP_THREADS`) to set the ONNX Runtime intra-op
+    thread count. Both default to 0, meaning auto. `intra_op_threads` configures the process-wide
+    inference session, so it only takes effect through the global options before the session is
+    created — worker pools with one process per core should set it to 1 to avoid thread
+    oversubscription (measured 1.7x higher total throughput on a 12-worker pool)
+- The cached text splitters are now keyed only on the chunking options (`max_tokens`, `overlap`,
+    `trim`). Previously every distinct combination of unrelated per-call options (e.g. `bits`, or
+    the batch size added above) retained an additional tokenizer-backed `TextSplitter` instance
+- Added the `ISCC_SCT_MODEL_DIR` environment variable to override the embedding model storage
+    directory (defaults to the platform-specific user data directory), enabling container images, CI
+    caches, and shared model directories (#19)
+- Fixed `__version__`, which still reported `0.2.0` in the 0.2.1 release. The version test now
+    compares against the installed package metadata so the two cannot drift apart silently
+
+## [0.2.1] - 2026-06-16
+
+- Disabled truncation on the tokenizer used for chunk sizing (new `chunking_tokenizer()`, separate
+    from the embedding `tokenizer()`). The vendored tokenizer truncates to 128 tokens, which made
+    the `tokenizers` >=0.23 chunk sizer emit one overflow encoding per 128 tokens, so sizing a large
+    probe string cost O(length) — the dominant cause of the issue #24 super-linear chunking. Sizing
+    the full text yields identical token counts and therefore identical chunk boundaries (verified
+    against `tests/chunking_vectors.json`) and unchanged ISCC codes, while further reducing the
+    guarded worst-case chunking time for PDF-extracted text. The token sizer is simplified
+    accordingly (no overflow summing; `count_nonpad_ids` removed)
+- Updated `semantic-text-splitter` to `>=0.32.0`, which adds the upstream "avoid sizing whole
+    distant split sections" fix (benbrandt/text-splitter#1184) and releases the GIL during native
+    chunking. The upstream fix probes lower-level semantic boundaries, so it speeds up texts with
+    distant but present separators; it does **not** cover spans with no intermediate separator at
+    all (the issue #24 PDF shape and spaceless CJK), where native chunking stays super-linear, so
+    the chunking guard is retained
+
+## [0.2.0] - 2026-06-14
+
+- Optimized the ONNX embedding model (`iscc-sct-v0.2.0.onnx`): the transformer graph is now fused
+    offline via `onnxruntime.transformers.optimizer` (attention, embedding layer norm, skip layer
+    norm, bias GELU). CPU inference is 1.25x-1.6x faster depending on hardware; generated ISCC codes
+    are unchanged (validated bit-identical at 256 bits across AVX2, AVX512-VNNI and CUDA on 1900+
+    chunks). The model weights are identical to v0.1.0 — only the graph structure changed
+- Fixed super-linear `split_text` runtime on texts without regular paragraph breaks (typical for
+    PDF-extracted text): such inputs now chunk via a guarded Python token sizer that skips
+    tokenizing huge splitter probes (114s → 7s for a 600KB book, #24). Chunk boundaries are
+    unchanged and now frozen by test vectors in `tests/chunking_vectors.json`
+- **BREAKING**: `onnxruntime` is no longer a base dependency — install `iscc-sct[cpu]` or
+    `iscc-sct[gpu]` instead. Previously the `[gpu]` extra was a silent no-op because the
+    unconditional CPU package shadowed `onnxruntime-gpu` (#23)
+- **BREAKING**: dropped Python 3.10 support and added Python 3.14 — `requires-python` is now
+    `>=3.11`. `onnxruntime` stopped shipping 3.10 wheels (last was 1.23.2) and added 3.14 wheels, so
+    the supported range tracks the runtime
+- **BREAKING**: renamed the CLI command from `sct` to `iscc-sct` (package name = CLI command is the
+    standard across all iscc projects); enables `uvx "iscc-sct[cpu]" <file>` one-liners
+- Added an `iscc-sct doctor` command that diagnoses the ONNX runtime (missing, or a CPU package
+    shadowing the GPU build), recommends the right `cpu`/`gpu` extra, and installs it on
+    confirmation
+- Import `onnxruntime` lazily on first model use, so `import iscc_sct` and the `iscc-sct` CLI load
+    without a runtime installed; the instructive `ImportError` now fires on first code generation
+- Warn at runtime when `onnxruntime-gpu` is installed but shadowed by the CPU package
+- Call `onnxruntime.preload_dlls()` before CUDA session creation so pip-provided NVIDIA libraries
+    are found without a system-wide CUDA install
+- Migrated project tooling from Poetry to [uv](https://docs.astral.sh/uv/) (uv_build backend)
+- Generate `requirements.txt` from `uv.lock` via `poe export-requirements`
+- Include LICENSE file in built distributions via `license-files`
+- CI: pin uv version, enforce `uv sync --locked`, verify `requirements.txt` matches `uv.lock`
+- CI: derive ONNX model cache directory from `iscc_sct.utils.MODEL_PATH`
+- CI: test matrix now covers Python 3.11–3.14 (dropped 3.10, added 3.14)
+- CI: added a release workflow that re-runs the full test matrix and publishes to PyPI on a
+    published GitHub Release, gated by a tag/version guard
+- Updated Hugging Face Space to Gradio 5.26.0 (matches locked version)
+- Fixed `format_yml` glob pattern that only worked on Windows
+- Updated dependencies (Gradio 6, pytest 9, pytest-cov 7, coverage 7.14)
+- Fixed CLI subprocess coverage measurement for pytest-cov 7 via coverage `patch = ["subprocess"]`
+- Adapted Gradio demo to Gradio 6: `theme` and `css` are now passed to `launch()`
+- Added [prek](https://github.com/j178/prek)-based pre-commit hooks (file hygiene checks, ruff
+    format/lint, mdformat)
+- Added a Zensical documentation site deployed to GitHub Pages at
+    [sct.iscc.codes](https://sct.iscc.codes), including a *For Coding Agents* reference page and
+    `llms.txt`/`llms-full.txt` for machine consumption
+
+## [0.1.4] - 2025-04-24
+
+- Added `bytes_offsets` option to generate UTF-8 byte positions instead of character positions
+- Updated dependencies
+
+## [0.1.3] - 2025-04-02
+
+- Update license, dependencies, and project metadata.
+- Update dependencies to the latest versions and workflow configurations (Poetry and GitHub
+    Actions).
+- Update CLI tests to use dynamic SCT command execution.
+- Add UTF-32BE chunk retrieval test for semantic text code.
+- Fix Python 3.13 support by removing Python 3.9 compatibility and updating version constraints.
+- Enhance documentation in demo.py.
 
 ## [0.1.2] - 2024-08-19
+
 - Encode granular features with base64
 - Refactor result format to generic ISCC data model
 - Add optional gradio GUI demo
 
 ## [0.1.1] - 2024-06-25
+
 - Handle text decoding errors gracefully
 - Handle feature bit-lengths independently
 - Improve model load time
 - Improve memory use with batched embedding
 
 ## [0.1.0] - 2024-06-25
+
 - Initial pre-release

LICENSE

@@ -1,10 +1,201 @@
-# LICENSE
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
 
-This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
 
-To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/4.0/ or send a letter to
-Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
+   1. Definitions.
 
-## Attribution
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
 
-Titusz Pan, ISCC Foundation - 2024
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2024-2026 Titusz Pan
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -1,12 +1,31 @@
 ---
-title: Iscc Sct
-emoji: 📚
-colorFrom: pink
-colorTo: green
+title: ISCC-LAB - Semantic-Code Text
+emoji: 🔮
+colorFrom: red
+colorTo: blue
 sdk: gradio
-sdk_version: 4.41.0
+sdk_version: 6.22.0
+python_version: "3.12"
 app_file: app.py
-pinned: false
+pinned: true
+license: cc-by-nc-sa-4.0
+short_description: Cross Lingual Similarity Preserving Text Simprints
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# ISCC Semantic Text-Code Demo
+
+Interactive demo for [iscc-sct](https://github.com/iscc/iscc-sct) — an experimental **Semantic-Code
+Text** implementation for the [ISCC](https://core.iscc.codes) (*International Standard Content
+Code*). It generates compact, similarity-preserving binary codes from text: semantically similar
+texts — including translations — produce codes with low hamming distance.
+
+Enter two texts (e.g. an original and its translation) to compare their Semantic Text-Codes,
+per-chunk simprints, and similarity.
+
+> **This is a proof of concept.** The algorithm is experimental and not part of the official
+> [ISO 24138:2024](https://www.iso.org/standard/77899.html) standard. Releases below v1.0.0 may
+> produce incompatible codes.
+
+- Documentation: [sct.iscc.codes](https://sct.iscc.codes)
+- Source code: [github.com/iscc/iscc-sct](https://github.com/iscc/iscc-sct)
+- Package: [pypi.org/project/iscc-sct](https://pypi.org/project/iscc-sct/)

app.py

@@ -1,4 +1,4 @@
-from iscc_sct.demo import demo
+from iscc_sct.demo import demo, launch_kwargs
 
 if __name__ == "__main__":
-    demo.launch()
+    demo.launch(**launch_kwargs)

docs/CNAME

@@ -0,0 +1 @@
+sct.iscc.codes

docs/conventions.md

@@ -0,0 +1,26 @@
+# Coding Convetions
+
+- Prefer httpx over requests for making http requests!
+- Write pragmatic, easily testable, and performant code!
+- Prefer short and pure functions where possible!
+- Keep the number of function arguments below 4!
+- Don´t use nested functions!
+- Write concise and to-the-point docstrings for all functions!
+- Write type comments style (PEP 484) instead of function annotations (PEP 3107)
+- Always add a correct PEP 484 style type comment as the first line after the function definition!
+- Use built-in collection types as generic types for annotations (PEP 585)!
+- Use the | (pipe) operator for writing union types (PEP 604)!
+
+Example function definition with (PEP 484) type comment and docstring:
+
+```python
+def tokenize_chunks(chunks, max_len=None):
+    # type: (list[str], int|None) -> dict
+    """
+    Tokenize text chunks into model-compatible formats.
+
+    :param chunks: Text chunks to tokenize.
+    :param max_len: Truncates chunks above max_len characters
+    :return: Dictionary of tokenized data including input IDs, attention masks, and type IDs.
+    """
+```

docs/explanation/how-it-works.md

@@ -0,0 +1,118 @@
+---
+icon: lucide/lightbulb
+description: How iscc-sct turns text into a similarity-preserving binary code, and why the codes match across languages.
+---
+
+# How it works
+
+This page explains how a Semantic Text-Code is built and why semantically similar texts — including
+translations — produce codes with low Hamming distance.
+
+## The problem
+
+The standard ISCC Content-Code for text matches on lexical similarity: it compares the words that
+appear in a document. That works well for near-duplicates, but it cannot tell that a German
+translation carries the same meaning as its English original — the two share almost no words.
+
+The Semantic Text-Code targets meaning instead of wording. It is built so that texts about the same
+thing land close together in code space, whatever language they are written in.
+
+## The pipeline
+
+```mermaid
+graph LR
+    A[Text] --> B[Split into chunks]
+    B --> C[Embed each chunk]
+    C --> D[Mean-pool to one vector]
+    D --> E[Binarize]
+    E --> F[ISCC header + base32]
+```
+
+Each stage has a specific job.
+
+### Split
+
+A document is first split into overlapping chunks at sensible boundaries (up to 127 tokens each,
+with up to 48 tokens of overlap). Overlap keeps a sentence that straddles a boundary from being lost
+to both chunks.
+
+Text without regular paragraph breaks — such as text extracted from print-layout PDFs — takes a
+guarded code path that produces the same chunks without the super-linear cost the naive splitter
+would incur on those inputs.
+
+### Embed
+
+Each chunk is run through a multilingual sentence-transformer model
+([paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2))
+exported to ONNX. The model turns a chunk into a 384-dimensional embedding: a vector of numbers that
+encodes the chunk's meaning. The model was trained so that texts with similar meaning — across more
+than 60 languages — map to nearby vectors.
+
+The per-token outputs are combined into one chunk vector by attention-mask pooling, then normalized.
+
+### Aggregate
+
+The chunk vectors are averaged into a single document vector and normalized again. This is the one
+vector that represents the whole document's meaning.
+
+### Binarize
+
+Each component of the document vector becomes one bit: positive values become `1`, the rest become
+`0`. The full vector yields a 384-bit digest.
+
+The digest is truncated to the requested length (`bits`, up to 256), prefixed with a 2-byte ISCC
+header that marks it as a Semantic Text-Code, and encoded as base32 with the `ISCC:` prefix.
+
+## Why binarized vectors still match
+
+Two texts with similar meaning produce similar embeddings — vectors pointing in nearly the same
+direction. Nearly-aligned vectors agree on the sign of most of their components, so their bit
+patterns agree on most bits, giving a low Hamming distance.
+
+Unrelated texts produce roughly perpendicular vectors, which agree on only about half their signs —
+so their codes differ in about half their bits. That gap between "few bits differ" and "about half
+the bits differ" is what makes a similarity threshold meaningful.
+
+| Property          | Behavior                                                |
+| ----------------- | ------------------------------------------------------- |
+| Similar meaning   | Few differing bits (low Hamming distance)               |
+| Unrelated content | About 50% of bits differ                                |
+| Translation       | Treated like similar meaning — a near-match             |
+| Longer `bits`     | Wider spread between near-matches and unrelated content |
+
+`iscc_distance()` measures similarity by stripping the `ISCC:` prefix and the 2-byte header, then
+counting the differing bits between the two code bodies. `cosine_similarity()` rescales that
+distance to a `-100`–`+100` score. See [comparing texts](../howto/compare-texts.md).
+
+## Cross-lingual matching
+
+Cross-lingual matching is a direct consequence of the embedding model. Because the model was trained
+to place a sentence and its translation near each other in vector space, the document vectors stay
+close, the sign patterns stay mostly equal, and the codes stay a few bits apart.
+
+No machine translation happens at code time. The shared meaning is captured once, by the model,
+during embedding.
+
+## Granular features
+
+Alongside the document code, `iscc-sct` can emit a simprint for each chunk. Because chunks carry
+their offsets, you can locate matching passages within documents and align similar passages across
+two documents — even when they sit at different positions. See
+[granular features](../howto/granular-features.md).
+
+## Relation to the ISCC standard
+
+The Semantic Text-Code is a planned, experimental ISCC-UNIT (MainType SEMANTIC, SubType TEXT). It is
+a proof of concept and **not** part of [ISO 24138:2024](https://www.iso.org/standard/77899.html).
+The codes it produces may change between releases below v1.0.0.
+
+The codes are still structurally compatible with the ISCC framework: a standard 2-byte ISCC header
+identifies the unit type and length, so a Semantic Text-Code slots in alongside the other
+ISCC-UNITs.
+
+## Further reading
+
+- **[Getting started](../tutorials/getting-started.md)** — Try the pipeline hands-on.
+- **[Compare texts](../howto/compare-texts.md)** — Use the distance in practice.
+- **[For Coding Agents](../reference/for-coding-agents.md)** — The exact algorithm, constants, and
+    invariants.

docs/howto/command-line.md

@@ -0,0 +1,123 @@
+---
+icon: lucide/terminal
+description: Generate Semantic Text-Codes from the command line, process files in bulk, run the runtime doctor, and launch the demo.
+---
+
+# Command line
+
+This guide covers the `iscc-sct` command-line tool: generating codes from files, processing many
+files at once, checking the ONNX runtime, and launching the demo.
+
+The CLI is installed with the package. Run `iscc-sct --help` to see every option:
+
+```text
+usage: iscc-sct [-h] [-b BITS] [-g] [-d] [-y] [path]
+
+Generate Semantic Text-Codes for text files.
+
+positional arguments:
+  path             Path to text files (glob patterns), 'doctor' to check the
+                   ONNX runtime, or 'gui' for the demo.
+
+options:
+  -h, --help       show this help message and exit
+  -b, --bits BITS  Bit-Length of Code (default 256)
+  -g, --granular   Activate granular processing.
+  -d, --debug      Show debugging messages.
+  -y, --yes        Auto-confirm the 'doctor' runtime install.
+```
+
+## Generate a code from a file
+
+Pass a path to print its Semantic Text-Code:
+
+```bash
+iscc-sct article.txt
+# ISCC:CADV3GG6JH3XEVRNSVYGCLJ7AAV3BOT5J7EHEZKPFXEGRJ2CTWACGZI
+```
+
+The CLI defaults to **256-bit** codes. Set a different length with `--bits`:
+
+```bash
+iscc-sct --bits 64 article.txt
+```
+
+!!! note "The CLI default differs from the library"
+
+    `iscc-sct` defaults to 256 bits, but `create()` in Python defaults to 64. Pass `--bits` (or the
+    `bits` argument) explicitly when the length matters. See [configuration](configuration.md).
+
+## Process multiple files
+
+The `path` argument accepts a glob pattern. Quote it so `iscc-sct` expands the pattern itself rather
+than the shell:
+
+```bash
+iscc-sct "texts/*.txt"
+```
+
+Each matching file prints its code. Files that cannot be decoded as UTF-8 are decoded with a
+detected character set, and empty files are skipped.
+
+## Granular output
+
+Add `--granular` to print the full metadata — document code plus per-chunk simprints, offsets,
+sizes, and contents — as JSON:
+
+```bash
+iscc-sct --granular article.txt
+```
+
+See [granular features](granular-features.md) for what the fields mean.
+
+## Run without installing
+
+With [`uv`](https://docs.astral.sh/uv/), run the CLI in one line without a permanent install:
+
+```bash
+uvx "iscc-sct[cpu]" article.txt
+```
+
+## Check the ONNX runtime
+
+`iscc-sct` needs exactly one ONNX runtime (`cpu` or `gpu` extra). The `doctor` command inspects your
+environment, names the problem, and recommends the right extra:
+
+```bash
+iscc-sct doctor
+```
+
+```text
+iscc-sct ONNX runtime check
+
+  ONNX runtime:   not installed
+  CUDA provider:  no
+  NVIDIA GPU:     no
+
+Status: no ONNX runtime installed.
+Recommended: pip install "iscc-sct[cpu]"
+```
+
+When a fix is available, `doctor` offers to run it. Add `--yes` to install without the prompt:
+
+```bash
+iscc-sct doctor --yes
+```
+
+`doctor` also detects the case where `onnxruntime-gpu` is installed but a plain `onnxruntime`
+package has shadowed it (issue #23), and reinstalls only the GPU build.
+
+## Launch the demo
+
+The `gui` command launches the interactive Gradio demo in your browser. It needs the `demo` extra:
+
+```bash
+pip install "iscc-sct[cpu,demo]"
+iscc-sct gui
+```
+
+## Related pages
+
+- **[Getting started](../tutorials/getting-started.md)** — Install and first code.
+- **[Configuration](configuration.md)** — Options and environment variables.
+- **[How it works](../explanation/how-it-works.md)** — What happens behind a code.

docs/howto/compare-texts.md

@@ -0,0 +1,122 @@
+---
+icon: lucide/git-compare
+description: Measure semantic and cross-lingual similarity between two texts using ISCC Semantic Text-Codes.
+---
+
+# Compare texts
+
+This guide shows how to measure how similar two texts are in meaning — including across languages —
+by comparing their Semantic Text-Codes.
+
+Similarity is computed from the codes, not the original text. You can store or share the codes and
+still compare content without keeping the source documents.
+
+## Compare two whole documents
+
+Generate a code for each text, then measure the Hamming distance between them with
+`iscc_distance()`. It returns the number of differing bits — lower means more similar:
+
+```python
+import iscc_sct as sct
+
+english = (
+    "An ISCC applies to a specific digital asset and is a data-descriptor deterministically "
+    "constructed from multiple hash digests using the algorithms and rules in this document. "
+    "This document does not provide information on registration of ISCCs."
+)
+german = (
+    "Ein ISCC bezieht sich auf ein bestimmtes digitales Gut und ist ein Daten-Deskriptor, der "
+    "deterministisch aus mehreren Hash-Digests unter Verwendung der Algorithmen und Regeln in "
+    "diesem Dokument erstellt wird. Dieses Dokument enthält keine Informationen über die "
+    "Registrierung von ISCCs."
+)
+
+a = sct.create(english)
+b = sct.create(german)
+
+print(sct.iscc_distance(a.iscc, b.iscc))
+# 3
+```
+
+`iscc_distance()` strips the `ISCC:` prefix and the 2-byte header, then compares the code bodies.
+The two codes must be the same bit-length, or it raises `ValueError`.
+
+## Read the distance
+
+Distance is measured in bits, so the scale depends on the code length. Compare a translation against
+an unrelated text at the same length to see the spread:
+
+```python
+import iscc_sct as sct
+
+unrelated = (
+    "The recipe calls for two cups of flour, a pinch of salt, and three ripe bananas mashed "
+    "until smooth before folding the mixture into the buttered baking tin."
+)
+u = sct.create(unrelated)
+
+print(sct.iscc_distance(a.iscc, b.iscc))  # translation
+# 3
+print(sct.iscc_distance(a.iscc, u.iscc))  # unrelated
+# 33
+```
+
+A translation sits a few bits apart; unrelated content sits near half the bit-length away (random
+codes differ in about 50% of their bits).
+
+## Choose a bit-length
+
+Longer codes spread similar and dissimilar texts further apart, which makes a threshold easier to
+pick. The same three texts at 256 bits:
+
+```python
+import iscc_sct as sct
+
+a256 = sct.create(english, bits=256)
+b256 = sct.create(german, bits=256)
+u256 = sct.create(unrelated, bits=256)
+
+print(sct.iscc_distance(a256.iscc, b256.iscc))  # translation
+# 23
+print(sct.iscc_distance(a256.iscc, u256.iscc))  # unrelated
+# 123
+```
+
+| Goal                                       | Suggested length |
+| ------------------------------------------ | ---------------- |
+| Compact storage, coarse matching           | 64 bits          |
+| Balanced precision                         | 128 bits         |
+| Finest discrimination between near-matches | 256 bits         |
+
+Both codes in a comparison must use the same `bits` value. See [configuration](configuration.md) for
+how to set the default.
+
+## Score similarity from raw digests
+
+`iscc_distance()` works on code strings. To compare raw digests directly — for example the output of
+`soft_hash_text_semantic()` — use `cosine_similarity()`, which scales the distance to a `-100` to
+`+100` score:
+
+```python
+import iscc_sct as sct
+
+a = sct.soft_hash_text_semantic("An ISCC applies to a specific digital asset.")
+b = sct.soft_hash_text_semantic("Ein ISCC bezieht sich auf ein bestimmtes digitales Gut.")
+
+print(sct.cosine_similarity(a, b))
+# 71
+```
+
+`hamming_distance()` returns the raw bit distance between two equal-length digests if you want the
+unscaled value. Both functions raise `ValueError` when the inputs differ in length.
+
+## Match passages instead of whole documents
+
+To find which parts of two documents are similar — rather than scoring them as a whole — use
+granular features and `granular_similarity()`. See [granular features](granular-features.md).
+
+## Related pages
+
+- **[Granular features](granular-features.md)** — Chunk-level matching with simprints.
+- **[How it works](../explanation/how-it-works.md)** — Why distance reflects meaning.
+- **[API reference](../reference/api.md)** — Signatures for every similarity function.

docs/howto/configuration.md

@@ -0,0 +1,126 @@
+---
+icon: lucide/settings
+description: Configure iscc-sct with per-call options, environment variables, or a global settings override.
+---
+
+# Configuration
+
+This guide shows how to control code length, chunking, and output detail through options — per call,
+through environment variables, or globally.
+
+Every option has the same name everywhere: as a keyword argument, as an `ISCC_SCT_*` environment
+variable, and as a field on the `SctOptions` model.
+
+## Options reference
+
+| Option             | Env variable                | Default | Notes                                                           |
+| ------------------ | --------------------------- | ------- | --------------------------------------------------------------- |
+| `bits`             | `ISCC_SCT_BITS`             | `64`    | Document code length. 32–256, multiple of 32.                   |
+| `bits_granular`    | `ISCC_SCT_BITS_GRANULAR`    | `64`    | Granular simprint length. 32–256, multiple of 32.               |
+| `characters`       | `ISCC_SCT_CHARACTERS`       | `True`  | Include the document character count.                           |
+| `embedding`        | `ISCC_SCT_EMBEDDING`        | `False` | Include the global document embedding vector.                   |
+| `precision`        | `ISCC_SCT_PRECISION`        | `8`     | Max fractional digits when storing the embedding.               |
+| `simprints`        | `ISCC_SCT_SIMPRINTS`        | `False` | Include granular per-chunk simprints.                           |
+| `offsets`          | `ISCC_SCT_OFFSETS`          | `False` | Include per-chunk offsets.                                      |
+| `byte_offsets`     | `ISCC_SCT_BYTE_OFFSETS`     | `False` | Report UTF-8 byte offsets instead of character offsets.         |
+| `sizes`            | `ISCC_SCT_SIZES`            | `False` | Include per-chunk sizes.                                        |
+| `contents`         | `ISCC_SCT_CONTENTS`         | `False` | Include the per-chunk text.                                     |
+| `max_tokens`       | `ISCC_SCT_MAX_TOKENS`       | `127`   | Max tokens per chunk. Cannot exceed 127.                        |
+| `overlap`          | `ISCC_SCT_OVERLAP`          | `48`    | Max tokens shared between adjacent chunks.                      |
+| `trim`             | `ISCC_SCT_TRIM`             | `False` | Trim whitespace from chunks.                                    |
+| `batch_size`       | `ISCC_SCT_BATCH_SIZE`       | `0`     | Chunks per inference batch. `0` = auto (1 on CPU, 100 on CUDA). |
+| `intra_op_threads` | `ISCC_SCT_INTRA_OP_THREADS` | `0`     | ONNX Runtime threads per operator. `0` = runtime default.       |
+
+The `granular=True` shortcut on `create()` is equivalent to setting `simprints`, `offsets`, `sizes`,
+and `contents` to `True` at once.
+
+The two inference options only change resource usage, never the generated codes. `intra_op_threads`
+configures the process-wide inference session, so it only takes effect through the global options or
+an environment variable before the first embedding call — worker pools with one process per core
+should set it to `1` to avoid thread oversubscription.
+
+## Override per call
+
+Pass options as keyword arguments to `create()`. They apply to that call only:
+
+```python
+import iscc_sct as sct
+
+text = "This is some sample text. It can be a longer document or even an entire book."
+meta = sct.create(text, bits=128, simprints=True, contents=True)
+print(meta.iscc)
+# ISCC:CABV3GG6JH3XEVRNSVYGCLJ7AAV3A
+```
+
+## Set defaults with environment variables
+
+Set any option through its `ISCC_SCT_*` variable. This changes the default for every call in the
+process:
+
+```bash
+export ISCC_SCT_BITS=128
+export ISCC_SCT_MAX_TOKENS=100
+```
+
+You can also place these in a `.env` file in the working directory — `iscc-sct` loads it
+automatically on import:
+
+```ini
+# .env
+ISCC_SCT_BITS=128
+ISCC_SCT_OVERLAP=24
+```
+
+## Override the global settings
+
+`sct_opts` is the global settings instance. Use `override()` to get a validated copy with some
+fields changed, without mutating the global:
+
+```python
+import iscc_sct as sct
+
+opts = sct.sct_opts.override({"bits": 128})
+print(opts.bits)  # 128
+print(sct.sct_opts.bits)  # 64 — the global is unchanged
+```
+
+Prefer `override()` or per-call keyword arguments over assigning to `sct_opts` fields directly. Both
+keep the global default predictable for other code in the same process.
+
+## Model storage directory
+
+The ~450 MB embedding model is downloaded on first use into the platform-specific user data
+directory. The `ISCC_SCT_MODEL_DIR` environment variable overrides that location — useful for
+container images, CI caches, or a model directory shared between users:
+
+```bash
+export ISCC_SCT_MODEL_DIR=/opt/shared/iscc-sct
+```
+
+Unlike the options above, `ISCC_SCT_MODEL_DIR` is not an `SctOptions` field — the path is resolved
+when `iscc_sct` is imported, so set it (or place it in `.env`) before importing.
+
+## Validation
+
+Options are validated whenever they are set. An out-of-range or wrong-shaped value raises a pydantic
+`ValidationError`:
+
+```python
+import iscc_sct as sct
+
+sct.sct_opts.override({"bits": 100})  # not a multiple of 32 -> ValidationError
+sct.sct_opts.override({"max_tokens": 200})  # exceeds 127 -> ValidationError
+```
+
+!!! warning "Library and CLI defaults differ"
+
+    `create()` and `SctOptions` default `bits` to **64**. The `iscc-sct` command-line tool defaults
+    `--bits` to **256**. The same text therefore yields a different code length depending on which entry
+    point you use. Set `bits` explicitly when you need a specific length.
+
+## Related pages
+
+- **[Compare texts](compare-texts.md)** — How `bits` affects matching.
+- **[Granular features](granular-features.md)** — `bits_granular`, offsets, and chunk contents.
+- **[Command line](command-line.md)** — Setting options from the CLI.
+- **[API reference](../reference/api.md)** — The `SctOptions` model in full.

docs/howto/granular-features.md

@@ -0,0 +1,156 @@
+---
+icon: lucide/layers
+description: Generate per-chunk simprints, choose offset and feature formats, reconstruct text, and match passages across documents.
+---
+
+# Granular features
+
+This guide shows how to work with granular features: the per-chunk fingerprints that let you match
+individual passages, locate them in the source text, and reconstruct content.
+
+A document code summarizes a whole text in one code. Granular features instead describe each chunk
+the text was split into, so you can compare documents passage by passage.
+
+## Generate granular features
+
+Pass `granular=True` to `create()`. Each chunk becomes a `Feature` with a simprint (its
+similarity-preserving fingerprint), a character `offset`, a `size`, and the chunk `content`:
+
+```python
+import iscc_sct as sct
+
+text = "This is some sample text. It can be a longer document or even an entire book."
+meta = sct.create(text, bits=256, granular=True)
+
+feature = meta.features[0].simprints[0]
+print(feature.simprint, feature.offset, feature.size)
+# XZjeSfdyVi0 0 77
+print(feature.content)
+# This is some sample text. It can be a longer document or even an entire book.
+```
+
+Short texts produce a single chunk. Longer texts produce many — see
+[matching passages](#match-passages-across-documents) below.
+
+## Choose a feature format
+
+Granular features come in two interchangeable shapes:
+
+| Format        | Shape                                                        | Produced by                |
+| ------------- | ------------------------------------------------------------ | -------------------------- |
+| Object-Format | a list of self-contained `Feature` objects                   | `create(granular=True)`    |
+| Index-Format  | parallel arrays: `simprints`, `offsets`, `sizes`, `contents` | `gen_text_code_semantic()` |
+
+`create()` returns Object-Format, which is easy to read and iterate. Convert between the two with
+`to_index_format()` and `to_object_format()`:
+
+```python
+import iscc_sct as sct
+
+meta = sct.create(text, bits=256, granular=True)  # Object-Format
+
+index = meta.to_index_format()
+print(index.features[0].simprints)  # ['XZjeSfdyVi0']
+print(index.features[0].offsets)  # [0]
+```
+
+Index-Format stores related attributes in parallel arrays, which is compact for storage and bulk
+indexing. The low-level `gen_text_code_semantic()` returns a plain dict already in Index-Format.
+
+## Control the simprint length
+
+Granular simprints have their own bit-length, set by `bits_granular` (default **64**). It is
+independent of the document `bits`, so `bits=256, granular=True` still yields 64-bit simprints:
+
+```python
+import iscc_sct as sct
+
+meta = sct.create(text, bits=256, granular=True, bits_granular=256)
+digest = sct.decode_base64(meta.features[0].simprints[0].simprint)
+print(len(digest) * 8)
+# 256
+```
+
+Shorter simprints save space; longer simprints discriminate finer between near-matching passages.
+
+## Reconstruct the original text
+
+When features include both `offset` and `content`, `get_content()` stitches the chunks back into the
+original text, removing the overlap between adjacent chunks:
+
+```python
+import iscc_sct as sct
+
+en = (
+    "The International Standard Content Code identifies digital content of any media type. "
+    "It is generated algorithmically from the content itself, much like a cryptographic hash. "
+    "Unlike a cryptographic hash, the ISCC preserves similarity between related items. "
+    "Two near-duplicate files therefore receive two codes that are close to each other. "
+    "This property supports deduplication, similarity clustering, and content provenance. "
+    "The Semantic Text-Code extends these ideas to the meaning of text across languages. "
+    "A translation of a document keeps a code that stays close to the original code. "
+    "That makes cross-lingual search and matching practical without machine translation."
+)
+meta = sct.create(en, granular=True)
+
+print(len(meta.features[0].simprints))  # number of chunks
+# 2
+print(meta.get_content() == en)
+# True
+```
+
+`get_overlaps()` returns the overlapping text between consecutive chunks if you need to inspect the
+chunk boundaries directly.
+
+## Use byte offsets
+
+Offsets and sizes are character positions by default. Enable `byte_offsets` to report UTF-8 byte
+positions instead — useful for fetching a chunk by random access from a remote file:
+
+```python
+import iscc_sct as sct
+
+meta = sct.create(en, granular=True, byte_offsets=True)
+print(meta.features[0].byte_offsets)  # True
+print(meta.features[0].simprints[1].offset)  # byte offset of the second chunk
+```
+
+## Match passages across documents
+
+`granular_similarity()` compares the simprints of two `Metadata` objects and returns the matching
+passages above a similarity threshold. Each result is a `(Feature, score, Feature)` tuple, where the
+score is the `cosine_similarity` between the two simprints:
+
+```python
+import iscc_sct as sct
+
+de = (
+    "Der International Standard Content Code identifiziert digitale Inhalte jedes Medientyps. "
+    "Er wird algorithmisch aus dem Inhalt selbst erzeugt, ähnlich einem kryptografischen Hash. "
+    "Anders als ein kryptografischer Hash bewahrt der ISCC die Ähnlichkeit verwandter Objekte. "
+    "Zwei nahezu identische Dateien erhalten daher zwei Codes, die nahe beieinander liegen. "
+    "Diese Eigenschaft unterstützt Deduplizierung, Ähnlichkeits-Clustering und Herkunftsnachweis. "
+    "Der Semantic Text-Code überträgt diese Ideen auf die Bedeutung von Text über Sprachen hinweg. "
+    "Eine Übersetzung eines Dokuments behält einen Code, der nahe am Originalcode bleibt. "
+    "Das macht sprachübergreifende Suche und Zuordnung ohne maschinelle Übersetzung praktikabel."
+)
+
+a = sct.create(en, granular=True)
+b = sct.create(de, granular=True)
+
+for feat_a, score, feat_b in sct.granular_similarity(a, b, threshold=80):
+    print(score, feat_a.offset, feat_b.offset)
+# 81 0 0
+# 90 340 356
+```
+
+Each English passage is matched to its closest German counterpart, even though the offsets differ
+because the languages produce text of different lengths. Raise `threshold` to keep only the
+strongest matches; lower it to surface weaker ones. Only the single best match per passage in the
+first document is returned.
+
+## Related pages
+
+- **[Compare texts](compare-texts.md)** — Whole-document similarity.
+- **[Configuration](configuration.md)** — Defaults for `bits_granular`, chunking, and offsets.
+- **[How it works](../explanation/how-it-works.md)** — How chunks and overlaps are produced.

docs/includes/abbreviations.md

@@ -0,0 +1,12 @@
+*[ISCC]: International Standard Content Code (ISO 24138)
+*[SCT]: Semantic Text-Code
+*[ISCC-UNIT]: A single self-describing component of an ISCC
+*[ONNX]: Open Neural Network Exchange
+*[CPU]: Central Processing Unit
+*[GPU]: Graphics Processing Unit
+*[CUDA]: Compute Unified Device Architecture (NVIDIA)
+*[MRL]: Matryoshka Representation Learning
+*[CLI]: Command-Line Interface
+*[API]: Application Programming Interface
+*[PDF]: Portable Document Format
+*[UTF-8]: 8-bit Unicode Transformation Format

docs/index.md

@@ -0,0 +1,157 @@
+---
+icon: lucide/house
+description: Cross-lingual, similarity-preserving semantic ISCC code for text content. Semantically similar texts, including translations, produce codes with low hamming distance.
+---
+
+# iscc-sct
+
+[![Tests](https://github.com/iscc/iscc-sct/actions/workflows/tests.yml/badge.svg)](https://github.com/iscc/iscc-sct/actions/workflows/tests.yml)
+[![Version](https://img.shields.io/pypi/v/iscc-sct.svg)](https://pypi.python.org/pypi/iscc-sct/)
+[![Downloads](https://pepy.tech/badge/iscc-sct)](https://pepy.tech/project/iscc-sct)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](https://github.com/iscc/iscc-sct/blob/main/LICENSE)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/iscc/iscc-sct)
+
+**A cross-lingual, similarity-preserving binary code for text. Semantically similar texts, including
+translations, produce codes with low hamming distance.**
+
+!!! warning "Proof of concept"
+
+    All releases below v1.0.0 may break backward compatibility and produce incompatible Semantic
+    Text-Codes. The algorithms in `iscc-sct` are experimental and **not** part of the official
+    [ISO 24138:2024](https://www.iso.org/standard/77899.html) standard.
+
+## Introduction
+
+The [ISCC](https://iscc.codes) framework already includes a Text-Code based on lexical similarity
+for near-duplicate matching. The Semantic Text-Code (SCT) is a planned additional ISCC-UNIT that
+captures a broader, more abstract similarity. It is engineered to be robust against rephrasing and,
+most notably, translations that lexical matching cannot detect.
+
+`iscc-sct` turns any text into a compact binary code built from a binarized, multilingual document
+embedding. The same content expressed in different languages maps to **(near-)identical codes**,
+opening up cross-lingual content identification and similarity detection.
+
+| Feature       | ISCC Content-Code Text   | ISCC Semantic-Code Text           |
+| ------------- | ------------------------ | --------------------------------- |
+| Focus         | Lexical similarity       | Semantic similarity               |
+| Cross-lingual | No                       | Yes                               |
+| Use case      | Near-duplicate detection | Semantic similarity, translations |
+
+**Key features:**
+
+- **Semantic similarity** - deep multilingual embeddings capture the meaning of the text
+- **Translation matching** - near-identical codes for the same content across languages
+- **Bit-length flexibility** - codes from 32 up to 256 bits for adjustable granularity
+- **ISCC compatible** - codes integrate directly with existing ISCC-based systems
+- **60+ languages** - one model covers a broad range of scripts and language pairs
+
+## Quick start
+
+=== "pip"
+
+    ```bash
+    pip install "iscc-sct[cpu]"
+    ```
+
+=== "uv"
+
+    ```bash
+    uv add "iscc-sct[cpu]"
+    ```
+
+=== "uvx (no install)"
+
+    ```bash
+    uvx "iscc-sct[cpu]" "path/to/textfile.txt"
+    ```
+
+Generate a Semantic Text-Code from Python:
+
+```python
+import iscc_sct as sct
+
+text = "This is some sample text. It can be a longer document or even an entire book."
+print(sct.create(text, bits=256).iscc)
+# ISCC:CADV3GG6JH3XEVRNSVYGCLJ7AAV3BOT5J7EHEZKPFXEGRJ2CTWACGZI
+```
+
+Or from the command line:
+
+```bash
+iscc-sct "path/to/textfile.txt"
+```
+
+!!! note "Choose exactly one ONNX runtime"
+
+    A plain `pip install iscc-sct` installs no ONNX runtime; the first code generation then fails with
+    install instructions. Install the `cpu` extra (works everywhere) or the `gpu` extra for NVIDIA CUDA
+    acceleration (requires CUDA 12.x and cuDNN 9.x), or run `iscc-sct doctor` to detect and install the
+    right one. Never install both: `onnxruntime` and `onnxruntime-gpu` unpack into the same directory
+    and silently overwrite each other.
+
+## How it works
+
+```text
+Text -> split -> embed -> mean-pool -> binarize -> ISCC header + base32
+```
+
+1. **Split** the text into overlapping chunks at syntactically sensible boundaries.
+1. **Embed** each chunk with a multilingual sentence-transformer running on ONNX.
+1. **Aggregate** the chunk embeddings into one mean-pooled, normalized document vector.
+1. **Binarize** the vector (positive components become 1-bits), truncate to the requested
+    bit-length, prefix the ISCC header, and base32-encode it.
+
+This process is robust to variation and translation, enabling cross-lingual matching from a short
+Simprint.
+
+## Documentation
+
+<div class="grid cards" markdown>
+
+- **[Getting started](tutorials/getting-started.md)** - Tutorial
+
+    Install, generate your first code, and match a translation across languages.
+
+- **[How-to guides](howto/compare-texts.md)** - Task recipes
+
+    Compare texts, work with granular features, configure options, and use the CLI.
+
+- **[How it works](explanation/how-it-works.md)** - Understand the design
+
+    The pipeline, cross-lingual matching, and why binarized vectors still match.
+
+- **[API reference](reference/api.md)** - Library details
+
+    Generated reference for `create()`, the data model, options, and similarity functions.
+
+- **[For Coding Agents](reference/for-coding-agents.md)** - Build on iscc-sct
+
+    A dense architecture map, constraints, and task recipes for AI coding agents.
+
+- **[Live Demo :lucide-external-link:](https://huggingface.co/spaces/iscc/iscc-sct)** - Try it in
+    the browser
+
+    Generate and compare Semantic Text-Codes interactively on Hugging Face Spaces.
+
+- **[Source Code :lucide-external-link:](https://github.com/iscc/iscc-sct)** - Read the
+    implementation
+
+    The full algorithm, CLI, and Gradio demo on GitHub.
+
+- **[Full text for LLMs :lucide-external-link:](llms-full.txt)** - Machine-readable docs
+
+    Every page concatenated into a single file for language models.
+
+</div>
+
+??? note "Supported languages (60+)"
+
+    Arabic, Armenian, Bengali, Bosnian, Bulgarian, Burmese, Catalan, Chinese (China), Chinese (Taiwan),
+    Croatian, Czech, Danish, Dutch, English, Estonian, Farsi, Finnish, French, French (Canada),
+    Galician, German, Greek, Gujarati, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Italian,
+    Japanese, Kannada, Korean, Kurdish, Latvian, Lithuanian, Macedonian, Malay, Malayalam, Marathi,
+    Mongolian, Norwegian Bokmål, Persian, Polish, Portuguese, Portuguese (Brazil), Romanian, Russian,
+    Serbian, Sinhala, Slovak, Slovenian, Spanish, Swedish, Tamil, Telugu, Thai, Turkish, Ukrainian,
+    Urdu, Vietnamese.
+
+[Source code on GitHub :lucide-external-link:](https://github.com/iscc/iscc-sct){ .md-button }

docs/javascripts/copilot.js

@@ -0,0 +1,27 @@
+/**
+ * Mount the ISCC-AI copilot chat widget with anonymous authentication.
+ *
+ * Shared across all ISCC documentation sites. Do not modify per project.
+ *
+ * Fetches a JWT token from the copilot-token endpoint, then mounts the
+ * Chainlit copilot widget with the token for cross-origin authentication.
+ */
+window.addEventListener("load", async function () {
+  if (typeof window.mountChainlitWidget !== "function") return;
+
+  var server = "https://iscc.ai";
+  var tokenUrl = server + "/api/copilot-token";
+
+  try {
+    var response = await fetch(tokenUrl);
+    var data = await response.json();
+    window.mountChainlitWidget({
+      chainlitServer: server,
+      theme: "light",
+      accessToken: data.accessToken,
+      customCssUrl: window.location.origin + "/stylesheets/copilot.css?v=7",
+    });
+  } catch (e) {
+    console.warn("ISCC-AI copilot: failed to fetch token", e);
+  }
+});

docs/javascripts/copypage.js

@@ -0,0 +1,200 @@
+/**
+ * "Copy page" split-button dropdown for documentation pages.
+ *
+ * Clicking the button copies the page as Markdown. Clicking the chevron
+ * opens a dropdown with additional actions: view as Markdown and edit on GitHub.
+ *
+ * Relies on per-page .md files generated by scripts/gen_llms_full.py.
+ */
+(function () {
+  "use strict";
+
+  var REPO_URL = "https://github.com/iscc/iscc-sct";
+  var EDIT_BRANCH = "main";
+
+  // Lucide icons (inline SVG)
+  var ICONS = {
+    copy:
+      '<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect width="14" height="14" x="8" y="8" rx="2" ry="2"/><path d="M4 16c-1.1 0-2-.9-2-2V4c0-1.1.9-2 2-2h10c1.1 0 2 .9 2 2"/></svg>',
+    fileText:
+      '<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M15 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V7Z"/><path d="M14 2v4a2 2 0 0 0 2 2h4"/><path d="M10 9H8"/><path d="M16 13H8"/><path d="M16 17H8"/></svg>',
+    pencil:
+      '<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21.174 6.812a1 1 0 0 0-3.986-3.987L3.842 16.174a2 2 0 0 0-.5.83l-1.321 4.352a.5.5 0 0 0 .623.622l4.353-1.32a2 2 0 0 0 .83-.497z"/><path d="m15 5 4 4"/></svg>',
+    chevronDown:
+      '<svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m6 9 6 6 6-6"/></svg>',
+    check:
+      '<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6 9 17l-5-5"/></svg>',
+  };
+
+  /** Derive the docs-relative path from the current page URL. */
+  function getDocPath() {
+    var path = window.location.pathname;
+    if (path.endsWith("/")) path = path.slice(0, -1);
+    return path ? path.slice(1) + ".md" : "index.md";
+  }
+
+  /** URL to the per-page .md file (generated by gen_llms_full.py). */
+  function getMarkdownUrl() {
+    return "/" + getDocPath();
+  }
+
+  /** GitHub edit URL for the current page source. */
+  function getEditUrl() {
+    return REPO_URL + "/edit/" + EDIT_BRANCH + "/docs/" + getDocPath();
+  }
+
+  /** Create a dropdown menu item button. */
+  function createItem(icon, title, desc, onClick) {
+    var btn = document.createElement("button");
+    btn.className = "copy-page__item";
+    btn.type = "button";
+    btn.innerHTML =
+      '<span class="copy-page__item-icon">' +
+      icon +
+      "</span>" +
+      '<span class="copy-page__item-text">' +
+      '<span class="copy-page__item-title">' +
+      title +
+      "</span>" +
+      '<span class="copy-page__item-desc">' +
+      desc +
+      "</span>" +
+      "</span>";
+    btn.addEventListener("click", function () {
+      onClick(btn);
+    });
+    return btn;
+  }
+
+  /** Copy the page Markdown to clipboard with visual feedback on the action button. */
+  function copyPage(el, actionBtn) {
+    el.classList.remove("copy-page--open");
+    var origHtml = actionBtn.innerHTML;
+    fetch(getMarkdownUrl())
+      .then(function (r) {
+        if (!r.ok) throw new Error(r.status);
+        return r.text();
+      })
+      .then(function (text) {
+        return navigator.clipboard.writeText(text);
+      })
+      .then(function () {
+        actionBtn.innerHTML = ICONS.check + "<span>Copied!</span>";
+        setTimeout(function () {
+          actionBtn.innerHTML = origHtml;
+        }, 2000);
+      })
+      .catch(function (err) {
+        console.error("Copy page failed:", err);
+      });
+  }
+
+  function init() {
+    var article = document.querySelector(".md-content__inner");
+    if (!article) return;
+    var h1 = article.querySelector("h1");
+    if (!h1) return;
+
+    // Container
+    var el = document.createElement("div");
+    el.className = "copy-page";
+
+    // Split button wrapper
+    var split = document.createElement("div");
+    split.className = "copy-page__split";
+
+    // Left: direct copy action
+    var actionBtn = document.createElement("button");
+    actionBtn.className = "copy-page__action";
+    actionBtn.type = "button";
+    actionBtn.title = "Copy page as Markdown for LLMs";
+    actionBtn.innerHTML = ICONS.copy + "<span>Copy page</span>";
+    actionBtn.addEventListener("click", function () {
+      copyPage(el, actionBtn);
+    });
+
+    // Right: chevron toggle for dropdown
+    var toggleBtn = document.createElement("button");
+    toggleBtn.className = "copy-page__toggle";
+    toggleBtn.type = "button";
+    toggleBtn.title = "More actions";
+    toggleBtn.innerHTML = ICONS.chevronDown;
+    toggleBtn.addEventListener("click", function () {
+      el.classList.toggle("copy-page--open");
+    });
+
+    split.appendChild(actionBtn);
+    split.appendChild(toggleBtn);
+
+    // Dropdown menu
+    var menu = document.createElement("div");
+    menu.className = "copy-page__menu";
+
+    // Item 1: Copy page as Markdown
+    menu.appendChild(
+      createItem(
+        ICONS.copy,
+        "Copy page",
+        "Copy page as Markdown for LLMs",
+        function () {
+          copyPage(el, actionBtn);
+        },
+      ),
+    );
+
+    // Item 2: View as Markdown
+    menu.appendChild(
+      createItem(
+        ICONS.fileText,
+        "View as Markdown",
+        "View this page as plain text",
+        function () {
+          el.classList.remove("copy-page--open");
+          window.open(getMarkdownUrl(), "_blank");
+        },
+      ),
+    );
+
+    // Item 2: Edit on GitHub
+    menu.appendChild(
+      createItem(
+        ICONS.pencil,
+        "Edit on GitHub",
+        "Edit this page on GitHub",
+        function () {
+          el.classList.remove("copy-page--open");
+          window.open(getEditUrl(), "_blank");
+        },
+      ),
+    );
+
+    el.appendChild(split);
+    el.appendChild(menu);
+
+    // Clicks inside the widget stay contained
+    el.addEventListener("click", function (e) {
+      e.stopPropagation();
+    });
+
+    // Wrap h1 and button in a flex container for alignment
+    var wrapper = document.createElement("div");
+    wrapper.className = "copy-page-heading";
+    h1.parentNode.insertBefore(wrapper, h1);
+    wrapper.appendChild(h1);
+    wrapper.appendChild(el);
+
+    // Close dropdown on outside click or Escape
+    document.addEventListener("click", function () {
+      el.classList.remove("copy-page--open");
+    });
+    document.addEventListener("keydown", function (e) {
+      if (e.key === "Escape") el.classList.remove("copy-page--open");
+    });
+  }
+
+  if (document.readyState === "loading") {
+    document.addEventListener("DOMContentLoaded", init);
+  } else {
+    init();
+  }
+})();

docs/llms.txt

@@ -0,0 +1,20 @@
+# iscc-sct
+
+> Cross-lingual, similarity-preserving semantic ISCC code for text content.
+
+iscc-sct generates the ISCC Semantic Text-Code: a compact binary code derived from multilingual
+text embeddings. Semantically similar texts - including translations - produce codes with low
+hamming distance, enabling cross-lingual similarity detection and near-duplicate matching.
+
+This is an experimental proof of concept and not part of the ISO 24138:2024 standard.
+
+Install: `pip install "iscc-sct[cpu]"`
+
+- [Source code](https://github.com/iscc/iscc-sct)
+- [PyPI](https://pypi.org/project/iscc-sct/)
+- [Full documentation](https://sct.iscc.codes/llms-full.txt)
+
+## Reference
+
+- [For Coding Agents](https://sct.iscc.codes/reference/for-coding-agents.md): Dense architecture
+  map, constraints, and task recipes for AI coding agents working on this codebase.

docs/overrides/main.html

@@ -0,0 +1,28 @@
+{% extends "base.html" %}
+
+{% block htmltitle %}
+  {% if page.title == config.site_name %}
+    <title>{{ config.site_name }} - Cross-Lingual Semantic Text-Code</title>
+  {% else %}
+    <title>{{ page.title }} - {{ config.site_name }}</title>
+  {% endif %}
+{% endblock %}
+
+{% block extrahead %}
+  <meta property="og:type" content="website">
+  <meta property="og:site_name" content="{{ config.site_name }}">
+  {% if page.title == config.site_name %}
+  <meta property="og:title" content="{{ config.site_name }} - Cross-Lingual Semantic Text-Code">
+  {% else %}
+  <meta property="og:title" content="{{ page.title }} - {{ config.site_name }}">
+  {% endif %}
+  <meta property="og:description" content="{% if page.meta and page.meta.description %}{{ page.meta.description }}{% else %}{{ config.site_description }}{% endif %}">
+  <meta property="og:url" content="{{ page.canonical_url }}">
+  <meta name="twitter:card" content="summary">
+  {% if page.title == config.site_name %}
+  <meta name="twitter:title" content="{{ config.site_name }} - Cross-Lingual Semantic Text-Code">
+  {% else %}
+  <meta name="twitter:title" content="{{ page.title }} - {{ config.site_name }}">
+  {% endif %}
+  <meta name="twitter:description" content="{% if page.meta and page.meta.description %}{{ page.meta.description }}{% else %}{{ config.site_description }}{% endif %}">
+{% endblock %}

docs/reference/api.md

@@ -0,0 +1,112 @@
+---
+icon: lucide/code
+description: Auto-generated reference for the public iscc-sct API - code generation, data model, options, and similarity functions.
+---
+
+# API reference
+
+This page documents the public API re-exported from the `iscc_sct` package. Import it with
+`import iscc_sct as sct`. The reference below is generated from the source docstrings.
+
+## High-level API
+
+The primary entry point. Returns a `Metadata` object with the code and, optionally, granular
+features.
+
+::: iscc_sct.create
+    options:
+      heading_level: 3
+
+## Core functions
+
+Lower-level building blocks of the processing pipeline.
+
+::: iscc_sct.gen_text_code_semantic
+    options:
+      heading_level: 3
+
+::: iscc_sct.code_text_semantic
+    options:
+      heading_level: 3
+
+::: iscc_sct.soft_hash_text_semantic
+    options:
+      heading_level: 3
+
+::: iscc_sct.embed_chunks
+    options:
+      heading_level: 3
+
+## Data model
+
+The result schema. `Metadata` holds the code and feature sets; `FeatureSet` and `Feature` carry the
+granular data in either Index-Format or Object-Format.
+
+::: iscc_sct.Metadata
+    options:
+      heading_level: 3
+
+::: iscc_sct.FeatureSet
+    options:
+      heading_level: 3
+
+::: iscc_sct.Feature
+    options:
+      heading_level: 3
+
+## Options
+
+The settings model. Configure it per call, with `ISCC_SCT_*` environment variables, or via
+`override()`.
+
+::: iscc_sct.SctOptions
+    options:
+      heading_level: 3
+
+## Similarity and distance
+
+Functions for comparing codes, digests, and granular features.
+
+::: iscc_sct.iscc_distance
+    options:
+      heading_level: 3
+
+::: iscc_sct.hamming_distance
+    options:
+      heading_level: 3
+
+::: iscc_sct.cosine_similarity
+    options:
+      heading_level: 3
+
+::: iscc_sct.granular_similarity
+    options:
+      heading_level: 3
+
+## Codecs and helpers
+
+Encoders, decoders, and utilities.
+
+::: iscc_sct.encode_base32
+    options:
+      heading_level: 3
+
+::: iscc_sct.decode_base32
+    options:
+      heading_level: 3
+
+::: iscc_sct.encode_base64
+    options:
+      heading_level: 3
+
+::: iscc_sct.decode_base64
+    options:
+      heading_level: 3
+
+::: iscc_sct.char_to_byte_offsets
+    options:
+      heading_level: 3
+
+::: iscc_sct.get_model
+    options:
+      heading_level: 3

docs/reference/for-coding-agents.md

@@ -0,0 +1,283 @@
+---
+icon: lucide/bot
+description: Dense, prescriptive reference for AI coding agents working on or integrating iscc-sct - architecture map, constraints, side effects, task recipes, change playbook, and common mistakes.
+---
+
+# For Coding Agents
+
+A compressed reference for AI agents working on **iscc-sct** (ISCC Semantic Text-Code). Read this
+before editing the codebase or integrating the library. Terminology matches the source exactly.
+
+`iscc-sct` turns text into a cross-lingual, similarity-preserving binary ISCC-UNIT (MainType
+SEMANTIC, SubType TEXT). It is an experimental proof of concept, **not** part of ISO 24138:2024.
+
+## Architecture map
+
+### File layout
+
+| Path                                  | Contains                                                                                                                                                                                                                                              |
+| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `iscc_sct/main.py`                    | `create()` - high-level API; returns a `Metadata` object in Object-Format.                                                                                                                                                                            |
+| `iscc_sct/code_semantic_text.py`      | Core algorithm. `gen_text_code_semantic()` (returns a plain dict, Index-Format), splitting, embedding, pooling, binarization. Module-level `@cache` singletons: `tokenizer()`, `chunking_tokenizer()`, `splitter()`, `splitter_guarded()`, `model()`. |
+| `iscc_sct/models.py`                  | Pydantic schema: `Feature`, `FeatureSet`, `Metadata` + format converters.                                                                                                                                                                             |
+| `iscc_sct/options.py`                 | `SctOptions` (pydantic-settings), `sct_opts` singleton, `.override()`.                                                                                                                                                                                |
+| `iscc_sct/utils.py`                   | Codecs (base32/base64url), distances, model download + blake3 integrity, `char_to_byte_offsets`, `MODEL_DIR`/`MODEL_PATH` (override dir via `ISCC_SCT_MODEL_DIR`), `timer`.                                                                           |
+| `iscc_sct/cli.py`                     | `iscc-sct` console entry point (`main()`): glob, charset detection, `gui` subcommand.                                                                                                                                                                 |
+| `iscc_sct/demo.py`, `iscc_sct/app.py` | Gradio demo (Hugging Face Space). Omitted from coverage.                                                                                                                                                                                              |
+| `iscc_sct/dev.py`                     | Dev-only poe task helpers. Omitted from coverage.                                                                                                                                                                                                     |
+| `iscc_sct/tokenizer.json`             | Vendored tokenizer. Byte-exact; excluded from whitespace/EOL hooks.                                                                                                                                                                                   |
+| `tests/chunking_vectors.json`         | Frozen chunk-boundary test vectors. Never hand-edit (see Change playbook).                                                                                                                                                                            |
+
+### Pipeline
+
+```text
+text -> split_text() -> embed_chunks() -> mean_pooling() -> binarize() -> ISCC header + base32
+```
+
+`gen_text_code_semantic(text, **options)` orchestrates this:
+
+1. Reject empty text (`ValueError`), then `sct_opts.override(options)`.
+1. `split_text()` -> `[(offset, chunk), ...]` at semantic boundaries (max 127 tokens, 48 overlap).
+1. `embed_chunks()` tokenizes in batches (auto: 1 on CPU, 100 on CUDA), runs the ONNX model,
+    `attention_pooling()` per chunk.
+1. `mean_pooling()` averages chunk vectors into one L2-normalized document vector.
+1. `binarize()` (`vec >= 0` -> bits), truncate to `bits // 8`, prefix the 2-byte ISCC header,
+    `encode_base32()`, prepend `"ISCC:"`.
+
+### Import flow
+
+```text
+iscc_sct/__init__.py  (star-imports, defines __version__)
+  -> options -> utils -> code_semantic_text -> models -> main
+main            -> models, code_semantic_text, options
+code_semantic_text -> onnxruntime (lazy import), semantic_text_splitter, tokenizers, numpy,
+                       iscc_sct as sct  (uses sct.* at call time, not import time)
+utils           -> models
+cli             -> main, charset_normalizer
+```
+
+### Public API
+
+`iscc_sct/__init__.py` re-exports every module's `__all__`. The public surface
+(`import iscc_sct as sct`):
+
+| Symbol                                                                          | Source             | Purpose                                          |
+| ------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------ |
+| `create(text, granular=False, **options)`                                       | main               | Primary API. Returns `Metadata` (Object-Format). |
+| `gen_text_code_semantic(text, **options)`                                       | code_semantic_text | Low-level. Returns `dict` (Index-Format).        |
+| `code_text_semantic(fp, **options)`                                             | code_semantic_text | Same, reading a UTF-8 file path.                 |
+| `soft_hash_text_semantic(text)`                                                 | code_semantic_text | Raw 384-bit digest (`bytes`), no header.         |
+| `embed_chunks(chunks, batch_size=None)`                                         | code_semantic_text | Chunk list -> embedding array.                   |
+| `Metadata`, `FeatureSet`, `Feature`                                             | models             | Result schema + converters.                      |
+| `SctOptions`, `sct_opts`                                                        | options            | Settings model + global instance.                |
+| `iscc_distance`, `hamming_distance`, `cosine_similarity`, `granular_similarity` | utils              | Similarity metrics.                              |
+| `encode_base32`, `decode_base32`, `encode_base64`, `decode_base64`              | utils              | Codecs.                                          |
+| `char_to_byte_offsets`, `get_model`, `MODEL_PATH`, `timer`                      | utils              | Helpers.                                         |
+| `__version__`                                                                   | `__init__`         | Package version string.                          |
+
+## Decision dispatch
+
+### Which entry point?
+
+| Goal                                                           | Use                                                     |
+| -------------------------------------------------------------- | ------------------------------------------------------- |
+| One ISCC code from a string, ready-to-use object               | `create(text)` -> `Metadata` (Object-Format)            |
+| Granular per-chunk features (simprints/offsets/sizes/contents) | `create(text, granular=True)`                           |
+| Compact parallel-array result for storage/indexing             | `gen_text_code_semantic(text, ...)` (Index-Format dict) |
+| Process a text file from disk                                  | `code_text_semantic(path)`                              |
+| Just the raw 384-bit vector digest                             | `soft_hash_text_semantic(text)` -> `bytes`              |
+| Command line / batch over files                                | `iscc-sct <glob>`                                       |
+
+### Which similarity metric?
+
+| Inputs                                | Use                                       | Returns                                 |
+| ------------------------------------- | ----------------------------------------- | --------------------------------------- |
+| Two full ISCC code strings            | `iscc_distance(iscc1, iscc2)`             | Hamming distance in bits (`int`)        |
+| Two raw digests of equal length       | `hamming_distance(a, b)`                  | bit distance                            |
+| Two raw digests, normalized score     | `cosine_similarity(a, b)`                 | `int` in `[-100, 100]`                  |
+| Two `Metadata` objects with simprints | `granular_similarity(a, b, threshold=80)` | `[(Feature, similarity, Feature), ...]` |
+
+### Which feature format?
+
+| Format        | Produced by                | Shape                                                    | Convert with                  |
+| ------------- | -------------------------- | -------------------------------------------------------- | ----------------------------- |
+| Index-Format  | `gen_text_code_semantic()` | parallel arrays `simprints`/`offsets`/`sizes`/`contents` | `Metadata.to_index_format()`  |
+| Object-Format | `create()`                 | list of self-contained `Feature` objects                 | `Metadata.to_object_format()` |
+
+## Constraints and invariants
+
+- **No base ONNX runtime.** The base package declares no `onnxruntime`. Install exactly one of the
+    mutually exclusive `cpu` / `gpu` extras. `onnxruntime` and `onnxruntime-gpu` unpack into the
+    same directory and clobber each other (issue #23); `tool.uv.conflicts` enforces exclusivity for
+    uv.
+- **ONNX runtime is imported lazily.** `import iscc_sct`, the `iscc-sct` CLI, and `iscc-sct doctor`
+    load without any runtime installed. `load_onnxruntime()` (called from `model()` on first code
+    generation) raises a guarded `ImportError` with install instructions - keep that guard intact.
+- **Codes are base32; simprints are base64url.** Never cross the codecs.
+- **ISCC header is fixed:** MainType `SEMANTIC` (`0001`), SubType `TEXT` (`0000`), Version `0000`,
+    plus a length nibble from `BIT_LEN_MAP`. 2 bytes, big-endian. `iscc_distance` strips the 5-char
+    `ISCC:` prefix, base32-decodes, drops the 2-byte header, then compares bodies of equal length.
+- **`bits` constraints:** `32 <= bits <= 256`, `multiple_of=32`. Same for `bits_granular`.
+- **Default bit-length differs by entry point:** `SctOptions.bits` / `create()` default to **64**;
+    the **CLI** `--bits` defaults to **256**.
+- **`SctOptions` validates on assignment.** Mutate only via `.override(dict)`, which deep-copies and
+    sets fields individually so validators run. Options flow as `**opts.model_dump()` through all
+    layers. Env vars use the `ISCC_SCT_` prefix; `.env` is loaded.
+- **Chunk boundaries are frozen** by `tests/chunking_vectors.json` and identical on both the normal
+    (`splitter`) and guarded (`splitter_guarded`) paths. `needs_split_guard()` routes texts whose
+    positions sit more than `SPLIT_GUARD_GAP` (8192) chars from the next paragraph separator to the
+    guarded sizer (PDF-extracted text, issue #24).
+- **Singletons:** `tokenizer()`, `chunking_tokenizer()`, `splitter(**opts)`,
+    `splitter_guarded(**opts)`, `model()` are `@cache`d. The splitter cache keys on the option
+    kwargs, so all option values must be hashable.
+- **Two tokenizers:** `tokenizer()` keeps the vendored truncation (128) + padding and embeds chunks;
+    `chunking_tokenizer()` disables truncation/padding and sizes chunks. Truncation on the sizer
+    makes `tokenizers` >=0.23 emit overflow encodings, the root cause of the issue #24 super-linear
+    chunking. Both yield identical boundaries, so don't merge them.
+- **Coverage must stay at 100%** (`--cov-fail-under=100`). GPU branches and model download use
+    `# pragma: no cover`. `dev.py`, `demo.py`, and `tests/` are omitted from coverage.
+- **Style:** PEP 484 **type comments** (first line after `def`), PEP 585 generics, PEP 604 unions.
+    Short pure functions, max 3 args, no nested functions, sphinx-style docstrings. Ruff line length
+    100, LF endings, rule F401 disabled.
+
+## Side effects catalog
+
+| Function                                                                  | Effect                                                                                                                                                                                                                  |
+| ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `model()` (first call)                                                    | If the model file is absent/corrupt, `get_model()` **downloads ~450 MB** to `MODEL_DIR` (platformdirs user-data dir, or `ISCC_SCT_MODEL_DIR`) and verifies a blake3 checksum. Creates a cached ONNX `InferenceSession`. |
+| `get_model()`                                                             | Network download + disk write to `MODEL_PATH`; blake3 integrity check (`# pragma: no cover`).                                                                                                                           |
+| `tokenizer()`, `chunking_tokenizer()`, `splitter()`, `splitter_guarded()` | Populate process-global `@cache` (loaded once).                                                                                                                                                                         |
+| `model()`                                                                 | Calls `rt.preload_dlls()` when CUDA is available; logs a warning via `warn_gpu_shadowed()` if `onnxruntime-gpu` is installed but CUDA is missing.                                                                       |
+| `import iscc_sct.options`                                                 | `load_dotenv()` reads `.env`; instantiates `sct_opts`.                                                                                                                                                                  |
+| `import iscc_sct.utils`                                                   | Resolves `MODEL_DIR` from `ISCC_SCT_MODEL_DIR` (else user data dir); `os.makedirs(MODEL_DIR)` (idempotent).                                                                                                             |
+| `create()`, `gen_text_code_semantic()`                                    | Pure given a loaded model: no disk writes, no mutation of inputs.                                                                                                                                                       |
+| CLI `main()`                                                              | Reads files, prints to stdout, removes the loguru logger unless `--debug`.                                                                                                                                              |
+
+## Task recipes
+
+### Generate a code (library)
+
+```python
+import iscc_sct as sct
+
+meta = sct.create("This is some sample text.", bits=256)
+print(meta.iscc)  # "ISCC:CAD..."
+print(meta.characters)  # input length in characters
+```
+
+### Granular per-chunk features
+
+```python
+import iscc_sct as sct
+
+meta = sct.create(long_text, bits=256, granular=True)
+for feature in meta.features[0].simprints:  # Object-Format
+    print(feature.offset, feature.size, feature.simprint, feature.content[:40])
+```
+
+### Compare two texts (cross-lingual)
+
+```python
+import iscc_sct as sct
+
+a = sct.create("An ISCC applies to a specific digital asset...")
+b = sct.create("Ein ISCC bezieht sich auf ein bestimmtes digitales Gut...")
+print(sct.iscc_distance(a.iscc, b.iscc))  # low bit distance => similar
+```
+
+### Match granular simprints
+
+```python
+import iscc_sct as sct
+
+a = sct.create(doc_a, granular=True)
+b = sct.create(doc_b, granular=True)
+for feat_a, similarity, feat_b in sct.granular_similarity(a, b, threshold=80):
+    print(similarity, feat_a.offset, feat_b.offset)
+```
+
+### Configure via options
+
+```python
+import iscc_sct as sct
+
+# Per-call override (validated copy of the global settings):
+meta = sct.create(text, bits=128, contents=True)
+
+# Or globally via environment: ISCC_SCT_BITS=128, ISCC_SCT_MAX_TOKENS=127, ...
+```
+
+## Change playbook
+
+| If you change...                                                                    | Also update...                                                                                                                                                         |
+| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| The embedding model file                                                            | `MODEL_FILENAME`, `MODEL_URL`, `MODEL_CHECKSUM` in `utils.py`; verify codes are bit-stable or bump version + mark BREAKING in CHANGELOG.                               |
+| Chunking (`max_tokens`, `overlap`, `split_text`, `needs_split_guard`, token sizing) | Regenerate `tests/chunking_vectors.json` via `uv run python tests/test_chunking_vectors.py` (only for an intentional algorithm change); multi-chunk codes shift.       |
+| A public function name/signature                                                    | The module's `__all__` (it is star-exported through `__init__.py`); README/docs examples.                                                                              |
+| Add a processing option                                                             | Add a field to `SctOptions` (env var name, constraints); it flows via `model_dump()`. Wire into `create()` if it is a granular toggle; add to the README config table. |
+| `Metadata` / `FeatureSet` / `Feature` schema                                        | Keep `to_index_format()`, `to_object_format()`, `get_content()`, `get_overlaps()` consistent.                                                                          |
+| ISCC header constants (`MAINTYPE`/`SUBTYPE`/`SCT_VERSION`/`BIT_LEN_MAP`)            | Codes change - this is BREAKING; bump version and document.                                                                                                            |
+| onnxruntime import / provider logic                                                 | Keep the `cpu`/`gpu` extras, `tool.uv.conflicts`, the `ImportError` guard, and `warn_gpu_shadowed()` in sync.                                                          |
+| Any code path                                                                       | Add tests to keep coverage at 100%; use `# pragma: no cover` only for GPU/download branches. Run `uv run poe all`.                                                     |
+| Dependencies                                                                        | Regenerate `requirements.txt` via `uv run poe export-requirements` (never hand-edit it).                                                                               |
+
+## Common mistakes
+
+**NEVER** add `onnxruntime` as a base dependency to "make install easier."
+
+```toml
+# WRONG - re-breaks issue #23: [gpu] then installs both wheels and CPU clobbers GPU
+dependencies = ["onnxruntime"]
+```
+
+**ALWAYS** keep it behind the mutually exclusive `cpu` / `gpu` extras.
+
+---
+
+**NEVER** use inline type annotations.
+
+```python
+def binarize(vec: NDArray) -> bytes: ...  # WRONG for this codebase
+```
+
+**ALWAYS** use PEP 484 type comments as the first line after `def`.
+
+```python
+def binarize(vec):
+    # type: (NDArray) -> bytes
+    ...
+```
+
+---
+
+**NEVER** expect Object-Format from the low-level function.
+
+```python
+meta = sct.gen_text_code_semantic(text)  # returns a dict in INDEX-Format
+meta.features[0].simprints  # WRONG - it is a dict, not Metadata
+```
+
+**ALWAYS** use `create()` for an Object-Format `Metadata`, or wrap the dict: `Metadata(**data)`.
+
+---
+
+**NEVER** mutate `SctOptions` fields directly or assume CLI/library defaults match.
+
+```python
+sct.sct_opts.bits = 256  # avoid - mutates the global; use .override()
+sct.create(text)  # library default bits=64, NOT the CLI's 256
+```
+
+**ALWAYS** pass overrides per call (`create(text, bits=256)`) or via `sct_opts.override({...})`.
+
+---
+
+**NEVER** hand-edit `tests/chunking_vectors.json`, `requirements.txt`, `iscc_sct/tokenizer.json`, or
+the model file. Each is generated or byte-exact.
+
+**ALWAYS** regenerate via the documented command and run `uv run poe all` before reporting done.
+
+---
+
+**NEVER** mix codecs: ISCC codes are RFC4648 base32 (no padding); granular simprints are base64url
+(no padding). Decode with the matching `decode_base32` / `decode_base64`.

docs/robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Allow: /

docs/stylesheets/copilot.css

@@ -0,0 +1,119 @@
+/* ISCC Copilot Widget Theme for Documentation Sites
+ *
+ * Shared across all ISCC documentation sites. Do not modify per project.
+ *
+ * Styles the Chainlit copilot chat widget to match the ISCC brand.
+ * Injected into the widget Shadow DOM via customCssUrl.
+ *
+ * Zensical sets html { font-size: 125% } (20px) which inflates all rem-based
+ * sizes inside the Chainlit widget by 25%. Instead of blanket zoom (which
+ * over-corrects body text), we pin Tailwind text utilities to their intended
+ * pixel values and zoom only the self-contained toggle button.
+ */
+
+/* Toggle button: self-contained element, zoom works cleanly here */
+#chainlit-copilot-button {
+  zoom: 0.8;
+}
+
+/* --- Fix Tailwind text utilities inflated by 125% root font-size ---
+ * Tailwind rem values resolve against html 20px instead of 16px.
+ * Pin each utility to its intended pixel size. */
+.text-xs  { font-size: 12px !important; line-height: 16px !important; }
+.text-sm  { font-size: 14px !important; line-height: 20px !important; }
+.text-base { font-size: 16px !important; line-height: 24px !important; }
+.text-lg  { font-size: 18px !important; line-height: 28px !important; }
+.text-xl  { font-size: 20px !important; line-height: 28px !important; }
+.text-2xl { font-size: 24px !important; line-height: 32px !important; }
+
+/* --- Widen the chat panel --- */
+.copilot-container-collapsed {
+  min-width: 420px !important;
+}
+
+/* Fix inflated rem-based spacing inside the chat panel.
+ * Scale factor: 0.8rem per intended 1rem (16/20 = 0.8). */
+[data-radix-popper-content-wrapper] {
+  font-size: 16px;
+}
+
+/* Ensure chat message body text renders at readable size */
+#chainlit-copilot-chat p,
+#chainlit-copilot-chat li,
+#chainlit-copilot-chat span:not([class]),
+#chainlit-copilot-chat div:not([class]) > span {
+  font-size: 15px !important;
+  line-height: 1.2 !important;
+}
+
+/* Force light theme variables - ISCC brand on white background.
+ * Target both .light and .dark to override regardless of server theme. */
+:root,
+:host,
+.light,
+.dark {
+  --background: 0 0% 100% !important;
+  --foreground: 213 69% 23% !important;
+  --primary: 356 89% 67% !important;
+  --primary-foreground: 0 0% 100% !important;
+  --secondary: 210 30% 95% !important;
+  --secondary-foreground: 213 69% 23% !important;
+  --accent: 210 30% 95% !important;
+  --accent-foreground: 213 69% 23% !important;
+  --muted: 210 20% 96% !important;
+  --muted-foreground: 213 40% 40% !important;
+  --card: 0 0% 100% !important;
+  --card-foreground: 213 69% 23% !important;
+  --popover: 0 0% 100% !important;
+  --popover-foreground: 213 69% 23% !important;
+  --border: 210 20% 85% !important;
+  --input: 0 0% 100% !important;
+  --ring: 211 100% 35% !important;
+}
+
+/* Floating toggle button - ISCC Coral */
+button.bg-primary.rounded-full,
+button[class*="bg-primary"][class*="rounded-full"] {
+  background-color: #f56169 !important;
+  background: #f56169 !important;
+}
+
+button.bg-primary.rounded-full:hover,
+button[class*="bg-primary"][class*="rounded-full"]:hover {
+  background-color: #e04550 !important;
+  background: #e04550 !important;
+}
+
+/* Send button and primary actions - ISCC Blue */
+button.bg-primary:not(.rounded-full),
+button[class*="bg-primary"]:not([class*="rounded-full"]),
+.bg-primary:not(.rounded-full) {
+  background-color: #0054b2 !important;
+  background: #0054b2 !important;
+}
+
+button.bg-primary:not(.rounded-full):hover,
+button[class*="bg-primary"]:not([class*="rounded-full"]):hover,
+.bg-primary:not(.rounded-full):hover {
+  background-color: #123663 !important;
+  background: #123663 !important;
+}
+
+/* Links - ISCC Blue */
+a {
+  color: #0054b2 !important;
+}
+
+/* Code blocks - ISCC navy */
+pre,
+pre code,
+.hljs {
+  background-color: #0d2847 !important;
+  color: #e6edf5 !important;
+}
+
+/* Inline code */
+code:not(pre code):not(.hljs) {
+  background-color: #e8ebef !important;
+  color: inherit !important;
+}

docs/stylesheets/extra.css

@@ -0,0 +1,289 @@
+/* ISCC Foundation Brand Colors
+ *
+ * Shared across all ISCC documentation sites. Do not modify per project.
+ *
+ * Override palette attribute selectors that Zensical sets by default.
+ * The [data-md-color-*] selectors match the specificity of the built-in
+ * palette CSS so our custom colors take precedence.
+ */
+
+/* Light mode: override primary (indigo -> ISCC Blue) */
+[data-md-color-primary="indigo"] {
+  --md-primary-fg-color: #0054b2;
+  --md-primary-fg-color--light: #4596f5;
+  --md-primary-fg-color--dark: #123663;
+  --md-primary-bg-color: #ffffff;
+  --md-primary-bg-color--light: #ffffffb3;
+}
+
+/* Light mode: override accent (indigo -> ISCC Blue) */
+[data-md-color-accent="indigo"] {
+  --md-accent-fg-color: #0054b2;
+  --md-accent-fg-color--transparent: #0054b21a;
+  --md-accent-bg-color: #ffffff;
+  --md-accent-bg-color--light: #ffffffb3;
+}
+
+/* Link color for light mode */
+[data-md-color-scheme="default"] {
+  --md-typeset-a-color: #0054b2;
+}
+
+/* Dark mode: override primary */
+[data-md-color-scheme="slate"][data-md-color-primary="indigo"] {
+  --md-primary-fg-color: #4596f5;
+  --md-primary-fg-color--light: #7ac2f7;
+  --md-primary-fg-color--dark: #0054b2;
+  --md-primary-bg-color: #123663;
+  --md-primary-bg-color--light: #12366380;
+  --md-typeset-a-color: #4596f5;
+}
+
+/* Dark mode: override accent */
+[data-md-color-scheme="slate"][data-md-color-accent="indigo"] {
+  --md-accent-fg-color: #4596f5;
+  --md-accent-fg-color--transparent: #4596f51a;
+}
+
+/* Dark mode: invert logo so the black logo becomes white */
+[data-md-color-scheme="slate"] .md-header__button.md-logo img,
+[data-md-color-scheme="slate"] .md-nav__button.md-logo img {
+  filter: invert(1);
+}
+
+/* ---------- Light mode: ISCC Blue header ---------- */
+[data-md-color-scheme="default"] .md-header {
+  background-color: #0054b2;
+  color: #ffffff;
+}
+
+[data-md-color-scheme="default"] .md-header .md-header__button,
+[data-md-color-scheme="default"] .md-header .md-header__topic,
+[data-md-color-scheme="default"] .md-header .md-header__title {
+  color: #ffffff;
+}
+
+[data-md-color-scheme="default"] .md-header .md-search__input::placeholder {
+  color: rgba(255, 255, 255, 0.7);
+}
+
+/* Search button: white text/icon on blue header */
+[data-md-color-scheme="default"] .md-header .md-search__button {
+  color: rgba(255, 255, 255, 0.8);
+  background-color: rgba(255, 255, 255, 0.12);
+  border-color: rgba(255, 255, 255, 0.3);
+}
+
+/* Search icon (mask uses background-color as fill) */
+[data-md-color-scheme="default"] .md-header .md-search__button::before {
+  background-color: rgba(255, 255, 255, 0.8);
+}
+
+/* Ctrl+K shortcut badge */
+[data-md-color-scheme="default"] .md-header .md-search__button::after {
+  background-color: #123663;
+  border-color: rgba(255, 255, 255, 0.3);
+  color: rgba(255, 255, 255, 0.8);
+}
+
+/* Invert logo in light mode header so it appears white on blue */
+[data-md-color-scheme="default"] .md-header__button.md-logo img {
+  filter: invert(1);
+}
+
+/* ---------- Dark mode: Deep Navy header ---------- */
+[data-md-color-scheme="slate"] .md-header {
+  background-color: #123663;
+  color: rgba(255, 255, 255, 0.9);
+}
+
+[data-md-color-scheme="slate"] .md-header .md-header__button,
+[data-md-color-scheme="slate"] .md-header .md-header__topic,
+[data-md-color-scheme="slate"] .md-header .md-header__title {
+  color: rgba(255, 255, 255, 0.9);
+}
+
+/* ---------- Footer: Deep Navy ---------- */
+[data-md-color-scheme="default"] .md-footer {
+  background-color: #123663;
+  color: rgba(255, 255, 255, 0.9);
+}
+
+[data-md-color-scheme="default"] .md-footer a {
+  color: rgba(255, 255, 255, 0.9);
+}
+
+[data-md-color-scheme="default"] .md-footer-meta {
+  background-color: rgba(0, 0, 0, 0.15);
+}
+
+[data-md-color-scheme="default"] .md-copyright,
+[data-md-color-scheme="default"] .md-copyright__highlight {
+  color: rgba(255, 255, 255, 0.7);
+}
+
+/* Match theme specificity: html .md-footer-meta.md-typeset a */
+html [data-md-color-scheme="default"] .md-footer-meta.md-typeset a {
+  color: rgba(255, 255, 255, 0.7);
+}
+
+/* Mermaid: softer edge color in dark mode */
+[data-md-color-scheme="slate"] .mermaid {
+  --md-mermaid-edge-color: #6e7681;
+}
+
+/* ---------- Copy page split-button dropdown ---------- */
+
+/* Flex wrapper pairs the h1 and the button on one line */
+.copy-page-heading {
+  display: flex;
+  align-items: flex-start;
+  gap: 0.8rem;
+}
+
+.copy-page-heading > h1 {
+  flex: 1;
+  min-width: 0;
+}
+
+.copy-page {
+  flex-shrink: 0;
+  position: relative;
+  margin-top: 0.35rem;
+  z-index: 2;
+}
+
+/* Split button: shared border wraps both halves */
+.copy-page__split {
+  display: inline-flex;
+  border: 1px solid var(--md-default-fg-color--lighter);
+  border-radius: 0.3rem;
+  overflow: hidden;
+  transition: border-color 0.2s;
+}
+
+.copy-page__split:hover {
+  border-color: var(--md-default-fg-color--light);
+}
+
+/* Left half: direct copy action */
+.copy-page__action {
+  display: inline-flex;
+  align-items: center;
+  gap: 0.3rem;
+  padding: 0.3rem 0.5rem;
+  border: none;
+  background: var(--md-default-bg-color);
+  color: var(--md-default-fg-color--light);
+  font-family: inherit;
+  font-size: 0.7rem;
+  line-height: 1.4;
+  cursor: pointer;
+  white-space: nowrap;
+  transition: background-color 0.15s, color 0.15s;
+}
+
+.copy-page__action:hover {
+  color: var(--md-default-fg-color);
+  background: var(--md-default-fg-color--lightest);
+}
+
+.copy-page__action svg {
+  flex-shrink: 0;
+}
+
+/* Right half: chevron dropdown toggle */
+.copy-page__toggle {
+  display: inline-flex;
+  align-items: center;
+  padding: 0.3rem 0.3rem;
+  border: none;
+  border-left: 1px solid var(--md-default-fg-color--lighter);
+  background: var(--md-default-bg-color);
+  color: var(--md-default-fg-color--light);
+  cursor: pointer;
+  transition: background-color 0.15s, color 0.15s;
+}
+
+.copy-page__toggle:hover {
+  color: var(--md-default-fg-color);
+  background: var(--md-default-fg-color--lightest);
+}
+
+/* Dropdown menu */
+.copy-page__menu {
+  display: none;
+  position: absolute;
+  top: calc(100% + 0.3rem);
+  right: 0;
+  min-width: 15rem;
+  background: var(--md-default-bg-color);
+  border: 1px solid var(--md-default-fg-color--lightest);
+  border-radius: 0.5rem;
+  box-shadow: 0 4px 16px rgba(0, 0, 0, 0.1);
+  overflow: hidden;
+  z-index: 10;
+}
+
+.copy-page--open .copy-page__menu {
+  display: block;
+}
+
+/* Menu items */
+.copy-page__item {
+  display: flex;
+  align-items: flex-start;
+  gap: 0.55rem;
+  width: 100%;
+  padding: 0.55rem 0.7rem;
+  border: none;
+  background: none;
+  color: var(--md-default-fg-color);
+  font-family: inherit;
+  font-size: 0.7rem;
+  line-height: 1.4;
+  cursor: pointer;
+  text-align: left;
+  transition: background-color 0.15s;
+}
+
+.copy-page__item:hover {
+  background: var(--md-default-fg-color--lightest);
+}
+
+.copy-page__item + .copy-page__item {
+  border-top: 1px solid var(--md-default-fg-color--lightest);
+}
+
+.copy-page__item-icon {
+  flex-shrink: 0;
+  margin-top: 0.1rem;
+  color: var(--md-default-fg-color--light);
+}
+
+.copy-page__item-text {
+  display: flex;
+  flex-direction: column;
+  gap: 0.05rem;
+}
+
+.copy-page__item-title {
+  font-weight: 600;
+}
+
+.copy-page__item-desc {
+  color: var(--md-default-fg-color--light);
+  font-size: 0.6rem;
+}
+
+/* Dark mode: stronger shadow */
+[data-md-color-scheme="slate"] .copy-page__menu {
+  box-shadow: 0 4px 16px rgba(0, 0, 0, 0.3);
+}
+
+/* Hide on small screens */
+@media screen and (max-width: 44.9375em) {
+  .copy-page {
+    display: none;
+  }
+}

docs/tutorials/getting-started.md

@@ -0,0 +1,138 @@
+---
+icon: lucide/rocket
+description: Install iscc-sct, generate your first Semantic Text-Code, and match a translation across languages.
+---
+
+# Getting started
+
+Install `iscc-sct`, generate your first Semantic Text-Code, and watch two languages produce
+near-identical codes.
+
+## Prerequisites
+
+- Python 3.10 or later
+- [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`
+
+## Install
+
+`iscc-sct` needs an ONNX runtime, selected through an install extra. The `cpu` extra works
+everywhere:
+
+=== "uv"
+
+    ```bash
+    uv add "iscc-sct[cpu]"
+    ```
+
+=== "pip"
+
+    ```bash
+    pip install "iscc-sct[cpu]"
+    ```
+
+!!! warning "Choose exactly one runtime"
+
+    Install the `cpu` extra, or the `gpu` extra for NVIDIA CUDA acceleration — never both. The
+    `onnxruntime` and `onnxruntime-gpu` packages unpack into the same directory and overwrite each
+    other. A plain `pip install iscc-sct` installs no runtime, and the first code generation then fails
+    with install instructions. If you are unsure which extra fits your machine, run `iscc-sct doctor`
+    (see the [command-line guide](../howto/command-line.md)).
+
+### Verify the installation
+
+```python
+import iscc_sct
+
+print(iscc_sct.__version__)
+```
+
+## Generate your first code
+
+Pass any text to `create()`. The first call downloads the embedding model (about 450 MB) to your
+user data directory; later calls reuse it.
+
+```python
+import iscc_sct as sct
+
+text = "This is some sample text. It can be a longer document or even an entire book."
+print(sct.create(text, bits=256).iscc)
+# ISCC:CADV3GG6JH3XEVRNSVYGCLJ7AAV3BOT5J7EHEZKPFXEGRJ2CTWACGZI
+```
+
+The `bits` argument sets the code length. Longer codes carry more detail; 64 bits is the library
+default, and 256 bits is the most precise. See [comparing texts](../howto/compare-texts.md) for how
+length affects matching.
+
+## Match a translation
+
+The point of a Semantic Text-Code is that meaning survives translation. Generate codes for the same
+passage in English and German, then measure their distance:
+
+```python
+import iscc_sct as sct
+
+english = (
+    "An ISCC applies to a specific digital asset and is a data-descriptor deterministically "
+    "constructed from multiple hash digests using the algorithms and rules in this document. "
+    "This document does not provide information on registration of ISCCs."
+)
+german = (
+    "Ein ISCC bezieht sich auf ein bestimmtes digitales Gut und ist ein Daten-Deskriptor, der "
+    "deterministisch aus mehreren Hash-Digests unter Verwendung der Algorithmen und Regeln in "
+    "diesem Dokument erstellt wird. Dieses Dokument enthält keine Informationen über die "
+    "Registrierung von ISCCs."
+)
+
+a = sct.create(english)
+b = sct.create(german)
+
+print(sct.iscc_distance(a.iscc, b.iscc))
+# 3
+```
+
+A distance of `3` bits out of 64 means the translation is a near-match. For contrast, an unrelated
+sentence lands far away:
+
+```python
+import iscc_sct as sct
+
+unrelated = (
+    "The recipe calls for two cups of flour, a pinch of salt, and three ripe bananas mashed "
+    "until smooth before folding the mixture into the buttered baking tin."
+)
+
+print(sct.iscc_distance(a.iscc, sct.create(unrelated).iscc))
+# 33
+```
+
+Low distance means similar meaning; high distance means unrelated content. Translation matching is
+the behavior that lexical (word-based) codes cannot provide.
+
+## Look inside the text
+
+Set `granular=True` to get per-chunk features alongside the document code. Each chunk carries its
+own offset, size, similarity-preserving fingerprint (simprint), and text:
+
+```python
+import iscc_sct as sct
+
+text = "This is some sample text. It can be a longer document or even an entire book."
+meta = sct.create(text, bits=256, granular=True)
+
+feature = meta.features[0].simprints[0]
+print(feature.offset, feature.size, feature.simprint)
+# 0 77 XZjeSfdyVi0
+```
+
+Granular features let you match individual passages across documents, even when the surrounding text
+differs. The [granular features guide](../howto/granular-features.md) covers chunk-level matching
+and content reconstruction.
+
+## Next steps
+
+- **[Compare texts](../howto/compare-texts.md)** — Measure semantic and cross-lingual similarity.
+- **[Granular features](../howto/granular-features.md)** — Work with per-chunk simprints and
+    offsets.
+- **[Configuration](../howto/configuration.md)** — Tune bit-length, chunking, and output options.
+- **[How it works](../explanation/how-it-works.md)** — Understand why the codes match across
+    languages.

iscc_sct/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.1.3"
+__version__ = "0.2.2"
 from iscc_sct.options import *
 from iscc_sct.utils import *
 from iscc_sct.code_semantic_text import *

iscc_sct/cli.py

@@ -1,5 +1,6 @@
 import argparse
 import glob
+import sys
 from pathlib import Path
 from loguru import logger
 from iscc_sct.main import create
@@ -7,11 +8,13 @@ from charset_normalizer import from_bytes
 
 
 def main():
-    parser = argparse.ArgumentParser(description="Generate Semantic Text-Codes for text files.")
+    parser = argparse.ArgumentParser(
+        prog="iscc-sct", description="Generate Semantic Text-Codes for text files."
+    )
     parser.add_argument(
         "path",
         type=str,
-        help="Path to text files (supports glob patterns) or 'gui' to launch Gradio demo.",
+        help="Path to text files (glob patterns), 'doctor' to check the ONNX runtime, or 'gui' for the demo.",
         nargs="?",
     )
     parser.add_argument(
@@ -21,6 +24,9 @@ def main():
         "-g", "--granular", action="store_true", help="Activate granular processing."
     )
     parser.add_argument("-d", "--debug", action="store_true", help="Show debugging messages.")
+    parser.add_argument(
+        "-y", "--yes", action="store_true", help="Auto-confirm the 'doctor' runtime install."
+    )
     args = parser.parse_args()
 
     if args.path is None:
@@ -32,15 +38,21 @@ def main():
 
     if args.path == "gui":  # pragma: no cover
         try:
-            from iscc_sct.demo import demo
+            from iscc_sct.demo import demo, launch_kwargs
 
-            demo.launch(inbrowser=True)
+            demo.launch(inbrowser=True, **launch_kwargs)
         except ImportError:
             print(
-                "Error: Gradio is not installed. Please install it with 'pip install gradio' to use the GUI."
+                "Error: Gradio is not installed. Please install it with "
+                "'pip install \"iscc-sct[demo]\"' to use the GUI."
             )
         return
 
+    if args.path == "doctor":
+        from iscc_sct.doctor import run_doctor
+
+        return run_doctor(assume_yes=args.yes)
+
     for path in glob.glob(args.path):
         path = Path(path)
         if path.is_file():
@@ -68,4 +80,4 @@ def main():
 
 
 if __name__ == "__main__":  # pragma: no cover
-    main()
+    sys.exit(main())

iscc_sct/code_semantic_text.py

@@ -20,15 +20,16 @@ The ISCC Text-Code Semantic is a content-based compact binary code generated fro
 """
 
 from loguru import logger as log
-from onnxruntime.capi.onnxruntime_pybind11_state import NoSuchFile
+
+from importlib.metadata import PackageNotFoundError, distribution
 from semantic_text_splitter import TextSplitter
 from tokenizers import Tokenizer
 from pathlib import Path
 from typing import Any
 import numpy as np
-import onnxruntime as rt
 from numpy.typing import NDArray
-from functools import cache
+from functools import cache, partial
+import re
 import iscc_sct as sct
 
 
@@ -59,6 +60,27 @@ MAINTYPE = "0001"  # SEMANTIC
 SUBTYPE = "0000"  # TEXT
 SCT_VERSION = "0000"  # V0
 
+# Newline runs that can act as paragraph-level split boundaries (2+ newline characters)
+NEWLINE_RUNS = re.compile(r"[\r\n]{2,}")
+
+# A single newline character - used by needs_split_guard to find separator-free spans
+NEWLINE = re.compile(r"[\r\n]")
+
+# Any Unicode whitespace - used by token_count_guarded to find a tokenizer word boundary.
+# The tokenizer's WhitespaceSplit pre-tokenizer splits on Unicode whitespace (not just ASCII),
+# so the prefix cut must match that to short-circuit NBSP/em-space/form-feed PDF text.
+WHITESPACE = re.compile(r"\s")
+
+# Max distance (chars) from any position to the next paragraph-level separator before
+# chunking switches to the guarded splitter (see needs_split_guard)
+SPLIT_GUARD_GAP = 8192
+
+# Chunks per inference batch chosen by resolve_batch_size when batch_size is 0 (auto).
+# On CPU one chunk per batch is both faster and leaner; on GPU large batches keep the device
+# saturated and the memory lives in VRAM (see resolve_batch_size).
+CPU_BATCH_SIZE = 1
+GPU_BATCH_SIZE = 100
+
 
 def code_text_semantic(fp, **options):
     # type: (Path|str, Any) -> dict[str, Any]
@@ -66,22 +88,24 @@ def code_text_semantic(fp, **options):
     Generate ISCC Semantic-Code Text from a text file.
 
     NOTE:
-        If you enable generating granular features with `features=True` those features will have
-        the same bit-length as the generated ISCC-UNIT.
-
-    :param fp: File path of plaintext file to process
-    :param options: Custom processing options for overriding global options
-    :key bits (int): Length of generated Semantic Text-Code in bits (default 64)
-    :key characters (bool): Return document character count (default True).
-    :key embedding (bool): Return global document embedding (default False).
-    :key precision (int): Max fractional digits for embeddings (default 8).
-    :key simprints (bool): Return granular document features (default False).
-    :key offsets (bool): Return character offsets for granular features (default False).
-    :key sizes (bool): Include sizes of granular features (number of chars, default False).
-    :key contents (bool): Return text chunks (default False).
-    :key max_tokens (int): Max tokens per chunk (default 127).
-    :key overlap (int): Max tokens allowed to overlap between chunks (default 48).
-    :key trim (int): Trim whitespace from chunks (default False).
+        Enable granular features with `simprints=True`. Their length is set by `bits_granular`
+        (default 64) and is independent of the document `bits`.
+
+    :param fp: File path of a plaintext file to process
+    :param options: Custom processing options for overriding global options. Recognized keys:
+
+        - ``bits`` (int): Length of generated Semantic Text-Code in bits (default 64).
+        - ``characters`` (bool): Return document character count (default True).
+        - ``embedding`` (bool): Return global document embedding (default False).
+        - ``precision`` (int): Max fractional digits for embeddings (default 8).
+        - ``simprints`` (bool): Return granular document features (default False).
+        - ``offsets`` (bool): Return character offsets for granular features (default False).
+        - ``sizes`` (bool): Include sizes of granular features in chars (default False).
+        - ``contents`` (bool): Return text chunks (default False).
+        - ``max_tokens`` (int): Max tokens per chunk (default 127).
+        - ``overlap`` (int): Max tokens allowed to overlap between chunks (default 48).
+        - ``trim`` (bool): Trim whitespace from chunks (default False).
+        - ``batch_size`` (int): Chunks per inference batch (default 0 = auto).
     :return: Dict with ISCC processing results
     """
     fp = Path(fp)
@@ -93,19 +117,26 @@ def gen_text_code_semantic(text, **options):
     """
     Create an ISCC Semantic-Code Text from plaintext.
 
-    :param str text: Plaint text for ISCC processing
-    :param options: Custom processing options for overriding global options
-    :key bits (int): Length of generated Semantic Text-Code in bits (default 64)
-    :key characters (bool): Return document character count (default True).
-    :key embedding (bool): Return global document embedding (default False).
-    :key precision (int): Max fractional digits for embeddings (default 8).
-    :key simprints (bool): Return granular document features (default False).
-    :key offsets (bool): Return character offsets for granular features (default False).
-    :key sizes (bool): Include sizes of granular features (number of chars, default False).
-    :key contents (bool): Return text chunks (default False).
-    :key max_tokens (int): Max tokens per chunk (default 127).
-    :key overlap (int): Max tokens allowed to overlap between chunks (default 48).
-    :key trim (int): Trim whitespace from chunks (default False).
+    :param str text: Plain text for ISCC processing
+    :param options: Custom processing options for overriding global options. Recognized keys:
+
+        - ``bits`` (int): Length of generated Semantic Text-Code in bits (default 64).
+        - ``characters`` (bool): Return document character count (default True).
+        - ``embedding`` (bool): Return global document embedding (default False).
+        - ``precision`` (int): Max fractional digits for embeddings (default 8).
+        - ``simprints`` (bool): Return granular document features (default False).
+        - ``offsets`` (bool): Return character offsets for granular features (default False).
+        - ``sizes`` (bool): Include sizes of granular features in chars (default False).
+        - ``contents`` (bool): Return text chunks (default False).
+        - ``max_tokens`` (int): Max tokens per chunk (default 127).
+        - ``overlap`` (int): Max tokens allowed overlapping between chunks (default 48).
+        - ``trim`` (bool): Trim whitespace from chunks (default False).
+        - ``batch_size`` (int): Chunks per inference batch (default 0 = auto).
+
+    NOTE:
+        `intra_op_threads` configures the shared inference session and therefore only takes
+        effect via the global options (`ISCC_SCT_INTRA_OP_THREADS`) before the session is
+        created. Passing it here has no effect.
     :return: Dict with ISCC processing results (using Index-Format for granular features)
     """
 
@@ -125,7 +156,7 @@ def gen_text_code_semantic(text, **options):
 
     # Chunk embedding
     with sct.timer("EMBEDDING time"):
-        embeddings = embed_chunks(chunks)
+        embeddings = embed_chunks(chunks, batch_size=opts.batch_size)
 
     # Create global document embedding
     embedding = mean_pooling(embeddings)
@@ -136,6 +167,8 @@ def gen_text_code_semantic(text, **options):
             "subtype": "text",
             "version": 0,
         }
+        if opts.offsets or opts.sizes:
+            feature_set["byte_offsets"] = opts.byte_offsets
         if opts.embedding:
             feature_set["embedding"] = compress(embedding, opts.precision)
         if opts.simprints:
@@ -144,7 +177,10 @@ def gen_text_code_semantic(text, **options):
         if opts.offsets:
             feature_set["offsets"] = offsets
         if opts.sizes:
-            feature_set["sizes"] = [len(chunk) for chunk in chunks]
+            if opts.byte_offsets:
+                feature_set["sizes"] = [len(chunk.encode("utf-8")) for chunk in chunks]
+            else:
+                feature_set["sizes"] = [len(chunk) for chunk in chunks]
         if opts.contents:
             feature_set["contents"] = chunks
         result["features"] = [feature_set]
@@ -160,7 +196,7 @@ def gen_text_code_semantic(text, **options):
 
 def soft_hash_text_semantic(text):
     # type: (str) -> bytes
-    """Creates a 256-bit semantic similarity preserving hash for text input."""
+    """Create a similarity-preserving hash for text as the full binarized document embedding (384 bits)."""
     chunks = [item[1] for item in split_text(text)]
     embeddings = embed_chunks(chunks)
     embedding = mean_pooling(embeddings)
@@ -176,12 +212,26 @@ def split_text(text, **options):
     :param text: Text to split.
     :param options: Custom processing options for overriding global options
     :key max_tokens (int): Max tokens per chunk (default 127).
-    :key overlap (int): Max tokens allowed to overlap between chunks (default 48).
+    :key overlap (int): Max tokens allowed overlapping between chunks (default 48).
     :key trim (int): Trim whitespace from chunks (default False).
-    :return: A list of offset, chunk tuples [(offset,chunk), ...]
+    :return: A list of offset, chunk tuples [(offset, chunk), ...]
     """
     opts = sct.sct_opts.override(options)
-    return splitter(**opts.model_dump()).chunk_indices(text)
+    select = splitter_guarded if needs_split_guard(text) else splitter
+    # Only the chunking options may reach the cached splitter constructors - unrelated knobs
+    # (e.g. batch_size) would fragment the cache with one TextSplitter per distinct value.
+    chunks = select(max_tokens=opts.max_tokens, overlap=opts.overlap, trim=opts.trim).chunk_indices(
+        text
+    )
+
+    if not opts.byte_offsets:
+        return chunks
+
+    # Convert character offsets to byte offsets
+    char_positions = [offset for offset, _ in chunks]
+    byte_positions = sct.char_to_byte_offsets(text, char_positions)
+
+    return [(byte_positions[i], chunk) for i, (_, chunk) in enumerate(chunks)]
 
 
 @cache
@@ -190,47 +240,281 @@ def tokenizer():
     """
     Load and cache the tokenizer model based on the predefined model name.
 
+    This tokenizer keeps the vendored truncation (128 tokens) and padding settings and is used
+    to embed chunks. For chunk sizing use chunking_tokenizer() instead.
+
     :return: An instance of the Tokenizer.
     """
     with sct.timer("TOKENIZER load time"):
         return Tokenizer.from_file(TOKENIZER_PATH.as_posix())
 
 
+@cache
+def chunking_tokenizer():
+    # type: () -> Tokenizer
+    """
+    Load and cache the tokenizer used for chunk sizing, with truncation and padding disabled.
+
+    The embedding tokenizer truncates to the model's 128-token window. With tokenizers >=0.23
+    the Hugging Face chunk sizer then sees one overflow encoding per 128 tokens, so sizing a
+    huge probe string costs O(length) and chunking degrades to super-linear runtime (issue
+    #24). Chunk sizing needs the true token count, so truncation is disabled here; padding
+    would only add tokens irrelevant to a count and is disabled too. Boundaries stay unchanged
+    because the splitter only compares sizes against the token capacity: at or below it the
+    full and truncated counts agree, and any larger probe exceeds it under both, so every
+    accept/reject decision is identical.
+
+    :return: A Tokenizer with truncation and padding disabled.
+    """
+    tok = Tokenizer.from_file(TOKENIZER_PATH.as_posix())
+    tok.no_truncation()
+    tok.no_padding()
+    return tok
+
+
 @cache
 def splitter(**options):
     # type: (Any) -> TextSplitter
     """
-    Load and cache the text splitter, initialized with tokenizer.
+    Load and cache the text splitter, initialized with the chunking tokenizer.
 
     :param options: Custom processing options for overriding global options
     :key max_tokens (int): Max tokens per chunk (default 127).
-    :key overlap (int): Max tokens allowed to overlap between chunks (default 48).
+    :key overlap (int): Max tokens allowed overlapping between chunks (default 48).
     :key trim (int): Trim whitespace from chunks (default False).
     :return: An instance of TextSplitter.
     """
     opts = sct.sct_opts.override(options)
     with sct.timer("TEXTSPLITTER load time"):
         return TextSplitter.from_huggingface_tokenizer(
-            tokenizer(), capacity=opts.max_tokens, overlap=opts.overlap, trim=opts.trim
+            chunking_tokenizer(), capacity=opts.max_tokens, overlap=opts.overlap, trim=opts.trim
         )
 
 
+@cache
+def splitter_guarded(**options):
+    # type: (Any) -> TextSplitter
+    """
+    Load and cache a text splitter that sizes chunks via a guarded Python callback.
+
+    Produces chunks identical to splitter() but avoids the super-linear cost of sizing huge
+    probe texts on inputs without regular paragraph separators (see needs_split_guard).
+
+    :param options: Custom processing options for overriding global options
+    :key max_tokens (int): Max tokens per chunk (default 127).
+    :key overlap (int): Max tokens allowed overlapping between chunks (default 48).
+    :key trim (int): Trim whitespace from chunks (default False).
+    :return: An instance of TextSplitter.
+    """
+    opts = sct.sct_opts.override(options)
+    sizer = partial(token_count_guarded, max_tokens=opts.max_tokens)
+    with sct.timer("TEXTSPLITTER load time"):
+        return TextSplitter.from_callback(
+            sizer, capacity=opts.max_tokens, overlap=opts.overlap, trim=opts.trim
+        )
+
+
+def needs_split_guard(text):
+    # type: (str) -> bool
+    """
+    Detect text where tokenizer-based chunking degrades to super-linear runtime.
+
+    To size an oversized section, text-splitter (>=0.32.0) probes prefixes up to that
+    section's lower-level semantic boundaries instead of tokenizing the whole section, which
+    keeps chunking near-linear while such boundaries exist. When a span carries no
+    intermediate separator for more than SPLIT_GUARD_GAP characters - a giant single
+    paragraph, a trailing separator-free run, or words sitting far from the next
+    paragraph-level separator (print-layout PDF extraction, issue #24) - the probe finds no
+    boundary and falls back to tokenizing the whole section, so chunking time grows
+    quadratically with the gap size.
+
+    Such texts are routed to the guarded splitter, whose token sizer caps that fallback cost.
+
+    :param text: Text to analyze.
+    :return: True if the guarded splitter should be used for this text.
+    """
+    runs = []
+    for match in NEWLINE_RUNS.finditer(text):
+        run = match.group()
+        level = len(run) - run.count("\r\n")  # number of newline graphemes in the run
+        if level >= 2:
+            runs.append((level, match.start(), match.end()))
+    for min_level in sorted({level for level, _, _ in runs}):
+        pos = 0
+        for level, start, end in runs:
+            if level < min_level:
+                continue
+            if start - pos > SPLIT_GUARD_GAP:
+                return True
+            pos = end
+    # A span containing no newline at all blows up the native sizer the same way: with no
+    # line- or paragraph-level separator to bound the probes, sizing reaches toward a distant
+    # or absent coarse separator. Catches giant single paragraphs and trailing separator-free
+    # runs that the paragraph-level scan above does not see.
+    pos = 0
+    for match in NEWLINE.finditer(text):
+        if match.start() - pos > SPLIT_GUARD_GAP:
+            return True
+        pos = match.end()
+    return len(text) - pos > SPLIT_GUARD_GAP
+
+
+def token_count(text):
+    # type: (str) -> int
+    """
+    Count tokens exactly like text-splitter's Hugging Face tokenizer chunk sizer.
+
+    Encodes without special tokens via the chunking tokenizer (truncation and padding
+    disabled), so the count reflects the full input text with no overflow encodings to sum.
+
+    :param text: Text to size.
+    :return: Number of tokens.
+    """
+    return len(chunking_tokenizer().encode(text, add_special_tokens=False).ids)
+
+
+def token_count_guarded(text, max_tokens):
+    # type: (str, int) -> int
+    """
+    Count tokens with a short-circuit for texts far larger than the chunk capacity.
+
+    For long texts, tokenize only a prefix that ends at a Unicode whitespace boundary. The
+    tokenizer pre-splits on whitespace (WhitespaceSplit + Metaspace), so the full text has at
+    least as many tokens as that prefix. If the prefix alone exceeds max_tokens, full
+    tokenization is skipped - the splitter only needs to know that the text is too big for one
+    chunk. The returned overestimate stays above max_tokens and grows with text length, so all
+    chunk capacity comparisons behave exactly as with real token counts.
+
+    :param text: Text to size.
+    :param max_tokens: Chunk capacity the splitter validates against.
+    :return: Number of tokens (exact, or an overestimate for oversized texts).
+    """
+    probe_chars = max_tokens * 10
+    if len(text) > probe_chars * 2:
+        window = text[:probe_chars]
+        cut = -1
+        for match in WHITESPACE.finditer(window):
+            cut = match.start()
+        if cut > 0:
+            prefix_count = token_count(text[:cut])
+            if prefix_count > max_tokens:
+                return prefix_count + len(text) - cut
+    return token_count(text)
+
+
+def warn_gpu_shadowed(available_providers):
+    # type: (list[str]) -> None
+    """
+    Warn when onnxruntime-gpu is installed but CUDA support is unavailable.
+
+    Both onnxruntime variant wheels unpack into the same directory, so installing the CPU
+    package alongside onnxruntime-gpu silently disables CUDA support.
+
+    :param available_providers: Providers reported by the installed onnxruntime build.
+    """
+    if "CUDAExecutionProvider" in available_providers:
+        return
+    try:
+        distribution("onnxruntime-gpu")
+    except PackageNotFoundError:
+        return
+    log.warning(
+        "onnxruntime-gpu is installed but CUDA support is unavailable - the onnxruntime CPU "
+        "package likely overwrote the GPU build. To fix run: pip uninstall -y onnxruntime "
+        'onnxruntime-gpu && pip install --force-reinstall "iscc-sct[gpu]"'
+    )
+
+
+ONNX_RUNTIME_MISSING = (
+    "iscc-sct requires an ONNX runtime. Install exactly one of:\n"
+    '  pip install "iscc-sct[cpu]"  # CPU inference\n'
+    '  pip install "iscc-sct[gpu]"  # NVIDIA CUDA accelerated inference'
+)
+
+
+def load_onnxruntime():
+    # type: () -> Any
+    """
+    Import and return the onnxruntime module.
+
+    The ONNX runtime is an optional dependency selected via the mutually exclusive cpu/gpu
+    extras, so the import is deferred until a model is actually needed. This lets the rest of
+    the package - and the `iscc-sct doctor` command - load without a runtime installed.
+
+    :return: The imported onnxruntime module.
+    :raises ImportError: If no ONNX runtime is installed.
+    """
+    try:
+        import onnxruntime as rt
+    except ImportError:  # pragma: no cover - exercised only without a runtime extra
+        raise ImportError(ONNX_RUNTIME_MISSING) from None
+    return rt
+
+
+def resolve_batch_size(batch_size, providers):
+    # type: (int, list[str]) -> int
+    """
+    Resolve the number of chunks to embed per inference batch.
+
+    A non-zero batch_size is used as given. Zero means auto, which depends on the execution
+    provider: on CPU the vendored tokenizer pads every batch to its longest chunk and attention
+    cost grows quadratically with sequence length, so batching mostly buys padding - single-chunk
+    batches are both faster and hold a smaller activation peak. A GPU instead needs large batches
+    to stay saturated, and its activations live in device memory rather than in the process.
+
+    :param batch_size: Configured batch size, 0 for auto.
+    :param providers: Execution providers the inference session actually runs on.
+    :return: Number of chunks per batch.
+    """
+    if batch_size:
+        return batch_size
+    return GPU_BATCH_SIZE if "CUDAExecutionProvider" in providers else CPU_BATCH_SIZE
+
+
+def session_options(rt, intra_op_threads):
+    # type: (Any, int) -> Any
+    """
+    Build the ONNX Runtime session options for the inference session.
+
+    Thread count is a session-level setting and the session is a process-wide singleton, so it
+    is read from the global options rather than passed per call. Leaving it at 0 keeps the
+    runtime default (one thread per core), which is fastest for a single process. Callers that
+    run a pool of one worker per core should set it to 1 to avoid oversubscription.
+
+    :param rt: The onnxruntime module.
+    :param intra_op_threads: Threads per operator, 0 for the runtime default.
+    :return: Configured onnxruntime.SessionOptions.
+    """
+    so = rt.SessionOptions()
+    so.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL
+    if intra_op_threads:
+        so.intra_op_num_threads = intra_op_threads
+    return so
+
+
 @cache
 def model():
-    # type: () -> rt.InferenceSession
+    # type: () -> Any
     """
     Load and cache the ONNX inference model from a specified path.
 
-    :return: An ONNX inference session.
+    :return: An ONNX inference session (onnxruntime.InferenceSession).
     """
+    rt = load_onnxruntime()
+    from onnxruntime.capi.onnxruntime_pybind11_state import NoSuchFile
+
     available_onnx_providers = rt.get_available_providers()
     log.debug(f"Available ONNX providers {', '.join(available_onnx_providers)}")
+    warn_gpu_shadowed(available_onnx_providers)
     selected_onnx_providers = ["CPUExecutionProvider"]
     if "CUDAExecutionProvider" in available_onnx_providers:  # pragma: no cover
         selected_onnx_providers.insert(0, "CUDAExecutionProvider")
+        if hasattr(rt, "preload_dlls"):
+            # Load CUDA/cuDNN libraries from pip-provided nvidia wheels or the system PATH
+            # before session creation (available since onnxruntime 1.21).
+            rt.preload_dlls()
     log.debug(f"Using ONNX providers {', '.join(selected_onnx_providers)}")
-    so = rt.SessionOptions()
-    so.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL
+    so = session_options(rt, sct.sct_opts.intra_op_threads)
     try:
         with sct.timer("ONNXMODEL load time"):
             return rt.InferenceSession(
@@ -259,14 +543,19 @@ def tokenize_chunks(chunks):
     return {"input_ids": input_ids, "attention_mask": attention_mask, "token_type_ids": type_ids}
 
 
-def embed_chunks(chunks, batch_size=100):
+def embed_chunks(chunks, batch_size=None):
+    # type: (list[str], int|None) -> NDArray
     """
     Embed text chunks and return vector embeddings.
 
     :param chunks: Text chunks to embed.
-    :param batch_size: Number of chunks to process in each batch.
+    :param batch_size: Number of chunks to process in each batch (None = use the global
+        `batch_size` option, 0 = auto, see resolve_batch_size).
     :return: An array of embeddings for each chunk.
     """
+    if batch_size is None:
+        batch_size = sct.sct_opts.batch_size
+    batch_size = resolve_batch_size(batch_size, model().get_providers())
     embeddings = []
     for start_idx in range(0, len(chunks), batch_size):
         batch_chunks = chunks[start_idx : start_idx + batch_size]
@@ -292,7 +581,7 @@ def embed_tokens(tokens):
 def attention_pooling(token_embeddings, attention_mask):
     # type: (np.array, np.array) -> np.array
     """
-    Apply attention mask based mean pooling to the token embeddings.
+    Apply attention mask-based mean pooling to the token embeddings.
 
     :param token_embeddings: Raw token embeddings from the model.
     :param attention_mask: Attention masks for the embeddings.
@@ -325,7 +614,7 @@ def binarize(vec):
     Binarize an embedding vector into a hash digest.
 
     :param vec: Vector to be binarized.
-    :return: A bytes object representing the binary hash.
+    :return: A byte object representing the binary hash.
     """
     return bytes((np.packbits(np.array(vec) >= 0)))
 
@@ -337,7 +626,7 @@ def compress(vec, precision):
 
     :param vec: Embedding vector.
     :param precision: Max number of fractional decimal places.
-    :return: Vector as native python list of rounded floats.
+    :return: Vector as a native python list of rounded floats.
     """
     rounded_array = np.around(vec, decimals=precision)
     compress_list = [round(x, precision) for x in rounded_array.tolist()]

iscc_sct/demo.py

@@ -3,6 +3,7 @@ Gradio demo showcasing ISCC Semantic Text Code.
 """
 
 from loguru import logger as log
+import binascii
 import gradio as gr
 import iscc_sct as sct
 import textwrap
@@ -68,9 +69,6 @@ def compute_iscc_code(text1, text2, bit_length):
     return code1["iscc"], code2["iscc"], similarity
 
 
-import binascii
-
-
 def compare_codes(code_a, code_b, bits):
     if code_a and code_b:
         code_a_str = code_a.value if hasattr(code_a, "value") else str(code_a)
@@ -140,7 +138,10 @@ iscc_theme = gr.themes.Default(
     radius_size=gr.themes.sizes.radius_none,
 )
 
-with gr.Blocks(css=custom_css, theme=iscc_theme) as demo:
+# Gradio 6 expects theme and css as launch() parameters - pass these at every launch site
+launch_kwargs = {"theme": iscc_theme, "css": custom_css}
+
+with gr.Blocks() as demo:
     with gr.Row(variant="panel"):
         gr.Markdown(
             """
@@ -519,4 +520,4 @@ For more information about the **ISCC** see:
         )
 
 if __name__ == "__main__":  # pragma: no cover
-    demo.launch()
+    demo.launch(**launch_kwargs)

iscc_sct/dev.py

@@ -24,7 +24,7 @@ def convert_lf():  # pragma: no cover
 
 
 def format_yml():
-    for f in HERE.glob("**\*.yml"):
+    for f in HERE.glob("**/*.yml"):
         with open(f, "rt", encoding="utf-8") as infile:
             data = yaml.safe_load(infile)
         with open(f, "wt", encoding="utf-8", newline="\n") as outf:

iscc_sct/doctor.py

@@ -0,0 +1,217 @@
+"""ONNX runtime diagnostics and opt-in repair for the `iscc-sct doctor` command.
+
+Inspects the installed ONNX runtime and classifies the environment: no runtime, a CPU
+`onnxruntime` package shadowing `onnxruntime-gpu` (both installed, CUDA disabled),
+`onnxruntime-gpu` installed without usable CUDA (driver/hardware issue, not fixable by
+reinstalling), or healthy. Recommends the right install extra and - only with explicit
+confirmation - runs the matching pip command in a subprocess.
+"""
+
+import importlib
+import shutil
+import subprocess
+import sys
+from importlib.metadata import PackageNotFoundError, distribution
+
+
+__all__ = ["doctor_report", "format_report", "run_doctor"]
+
+
+def _dist_installed(name):
+    # type: (str) -> bool
+    """
+    Check whether a distribution is installed.
+
+    :param name: Distribution name to look up.
+    :return: True if the distribution is installed.
+    """
+    try:
+        distribution(name)
+        return True
+    except PackageNotFoundError:
+        return False
+
+
+def _onnx_providers():
+    # type: () -> list[str] | None
+    """
+    Return the available onnxruntime execution providers.
+
+    :return: Provider names reported by onnxruntime, or None if no runtime is importable.
+    """
+    try:
+        rt = importlib.import_module("onnxruntime")
+    except ImportError:
+        return None
+    return rt.get_available_providers()
+
+
+def _gpu_present():
+    # type: () -> bool
+    """
+    Detect an NVIDIA GPU by probing for the nvidia-smi tool on PATH.
+
+    :return: True if nvidia-smi is found on PATH.
+    """
+    return shutil.which("nvidia-smi") is not None
+
+
+def _diagnose(providers, gpu_dist, gpu_present, cpu_dist):
+    # type: (list[str]|None, bool, bool, bool) -> dict
+    """
+    Derive runtime status, recommended extra, and pip fix from environment facts.
+
+    :param providers: onnxruntime.get_available_providers() result, or None if not importable.
+    :param gpu_dist: Whether the onnxruntime-gpu distribution is installed.
+    :param gpu_present: Whether an NVIDIA GPU was detected.
+    :param cpu_dist: Whether the CPU onnxruntime distribution is installed.
+    :return: Report dict with status, runtime, cuda, gpu_present, gpu_dist, cpu_dist, extra,
+        fix, hint.
+    """
+    runtime = providers is not None
+    cuda = runtime and "CUDAExecutionProvider" in providers
+    extra = "gpu" if gpu_present else "cpu"
+
+    if runtime and gpu_dist and cpu_dist and not cuda:
+        # Both runtime distributions are installed: their wheels unpacked into the same
+        # directory and the CPU build clobbered onnxruntime-gpu, disabling CUDA (issue #23).
+        status = "shadowed"
+        fix = [
+            ["uninstall", "-y", "onnxruntime", "onnxruntime-gpu"],
+            ["install", "--force-reinstall", "iscc-sct[gpu]"],
+        ]
+    elif not runtime:
+        status = "missing"
+        fix = [["install", f"iscc-sct[{extra}]"]]
+    elif gpu_dist and not cuda:
+        # onnxruntime-gpu is installed and importable but CUDA is unavailable, and nothing
+        # shadowed it - the NVIDIA driver/GPU is missing or the CUDA libraries failed to load.
+        # Reinstalling cannot fix an environment problem, so offer no pip fix.
+        status = "no_cuda"
+        fix = []
+    else:
+        status = "ok"
+        fix = []
+
+    hint = status == "ok" and gpu_present and not cuda and not gpu_dist
+    return {
+        "status": status,
+        "runtime": runtime,
+        "cuda": cuda,
+        "gpu_present": gpu_present,
+        "gpu_dist": gpu_dist,
+        "cpu_dist": cpu_dist,
+        "extra": extra,
+        "fix": fix,
+        "hint": hint,
+    }
+
+
+def doctor_report():
+    # type: () -> dict
+    """
+    Diagnose the ONNX runtime in the current environment.
+
+    :return: A report dict (see `_diagnose`).
+    """
+    return _diagnose(
+        _onnx_providers(),
+        _dist_installed("onnxruntime-gpu"),
+        _gpu_present(),
+        _dist_installed("onnxruntime"),
+    )
+
+
+def _yn(flag):
+    # type: (bool) -> str
+    """Render a boolean as yes/no."""
+    return "yes" if flag else "no"
+
+
+def _runtime_label(report):
+    # type: (dict) -> str
+    """Describe which ONNX runtime package(s) are installed."""
+    if report["gpu_dist"] and report["cpu_dist"]:
+        return "onnxruntime + onnxruntime-gpu (conflicting)"
+    if report["gpu_dist"]:
+        return "onnxruntime-gpu"
+    if report["runtime"]:
+        return "onnxruntime"
+    return "not installed"
+
+
+def format_report(report):
+    # type: (dict) -> str
+    """
+    Render a human-readable diagnosis.
+
+    :param report: Report dict from `doctor_report`.
+    :return: Multi-line report string.
+    """
+    lines = [
+        "iscc-sct ONNX runtime check",
+        "",
+        f"  ONNX runtime:   {_runtime_label(report)}",
+        f"  CUDA provider:  {_yn(report['cuda'])}",
+        f"  NVIDIA GPU:     {_yn(report['gpu_present'])}",
+        "",
+    ]
+    if report["status"] == "ok":
+        lines.append("Status: OK - a working ONNX runtime is installed.")
+        if report["hint"]:
+            lines.append('GPU detected. For CUDA acceleration: pip install "iscc-sct[gpu]"')
+    elif report["status"] == "missing":
+        lines.append("Status: no ONNX runtime installed.")
+        lines.append(f'Recommended: pip install "iscc-sct[{report["extra"]}]"')
+    elif report["status"] == "no_cuda":
+        lines.append(
+            "Status: onnxruntime-gpu is installed and runs on CPU, but CUDA is unavailable - no "
+            "NVIDIA driver/GPU was detected or the CUDA libraries failed to load. This is an "
+            "environment issue, not a packaging problem; reinstalling will not help."
+        )
+    else:
+        lines.append(
+            "Status: onnxruntime-gpu is installed but CUDA is unavailable - the CPU onnxruntime "
+            "package shadowed the GPU build. The fix reinstalls only the GPU build."
+        )
+    return "\n".join(lines)
+
+
+def _confirm(assume_yes):
+    # type: (bool) -> bool
+    """
+    Confirm the fix with the user unless auto-confirmed.
+
+    :param assume_yes: Skip the prompt and proceed when True.
+    :return: True to proceed with the fix.
+    """
+    if assume_yes:
+        return True
+    try:
+        answer = input("Install it now? [y/N] ")
+    except EOFError:
+        return False
+    return answer.strip().lower() in ("y", "yes")
+
+
+def run_doctor(assume_yes=False):
+    # type: (bool) -> int
+    """
+    Print the ONNX runtime diagnosis and, on confirmation, run the recommended pip fix.
+
+    :param assume_yes: Auto-confirm the fix without prompting.
+    :return: Exit code: 0 when healthy or the fix ran cleanly, 1 otherwise.
+    """
+    report = doctor_report()
+    print(format_report(report))
+    if not report["fix"]:
+        return 0
+    if not _confirm(assume_yes):
+        print("No changes made.")
+        return 1
+    failed = False
+    for args in report["fix"]:
+        print(f"  running: pip {' '.join(args)}")
+        completed = subprocess.run([sys.executable, "-m", "pip", *args])
+        failed = failed or completed.returncode != 0
+    return 1 if failed else 0

iscc_sct/models.py

@@ -98,6 +98,7 @@ class FeatureSet(PrettyBaseModel):
     maintype: str = "semantic"
     subtype: str = "text"
     version: int = 0
+    byte_offsets: Optional[bool] = False
     embedding: Optional[List[float]] = None
     simprints: Optional[
         Union[

iscc_sct/options.py

@@ -47,6 +47,11 @@ class SctOptions(BaseSettings):
         False, description="ISCC_SCT_OFFSETS - Include offsets of granular features"
     )
 
+    byte_offsets: bool = Field(
+        False,
+        description="ISCC_SCT_BYTE_OFFSETS - Use UTF-8 byte offsets instead of character offsets",
+    )
+
     sizes: bool = Field(
         False, description="ISCC_SCT_SIZES - Include sizes of granular features (number of chars)"
     )
@@ -68,6 +73,19 @@ class SctOptions(BaseSettings):
         False, description="ISCC_SCT_TRIM - Trim whitespace from chunks (Default False)"
     )
 
+    batch_size: int = Field(
+        0,
+        description="ISCC_SCT_BATCH_SIZE - Chunks per inference batch (0 = auto, Default 0)",
+        ge=0,
+    )
+
+    intra_op_threads: int = Field(
+        0,
+        description="ISCC_SCT_INTRA_OP_THREADS - ONNX Runtime threads per operator "
+        "(0 = runtime default, Default 0)",
+        ge=0,
+    )
+
     model_config = SettingsConfigDict(
         env_file=".env",
         env_file_encoding="utf-8",

iscc_sct/utils.py

@@ -15,7 +15,11 @@ from iscc_sct.models import Metadata, Feature
 APP_NAME = "iscc-sct"
 APP_AUTHOR = "iscc"
 dirs = PlatformDirs(appname=APP_NAME, appauthor=APP_AUTHOR)
-os.makedirs(dirs.user_data_dir, exist_ok=True)
+# ISCC_SCT_MODEL_DIR overrides the platform-specific model storage directory (issue #19).
+# Read via os.environ (not SctOptions) because the path must be known at import time; a `.env`
+# file still works because `iscc_sct.options` runs `load_dotenv()` before this module is imported.
+MODEL_DIR = Path(os.environ.get("ISCC_SCT_MODEL_DIR") or dirs.user_data_dir)
+os.makedirs(MODEL_DIR, exist_ok=True)
 
 
 __all__ = [
@@ -29,16 +33,17 @@ __all__ = [
     "iscc_distance",
     "cosine_similarity",
     "granular_similarity",
+    "char_to_byte_offsets",
     "MODEL_PATH",
 ]
 
 
 BASE_VERSION = "1.0.0"
 BASE_URL = f"https://github.com/iscc/iscc-binaries/releases/download/v{BASE_VERSION}"
-MODEL_FILENAME = "iscc-sct-v0.1.0.onnx"
+MODEL_FILENAME = "iscc-sct-v0.2.0.onnx"
 MODEL_URL = f"{BASE_URL}/{MODEL_FILENAME}"
-MODEL_PATH = Path(dirs.user_data_dir) / MODEL_FILENAME
-MODEL_CHECKSUM = "ff254d62db55ed88a1451b323a66416f60838dd2f0338dba21bc3b8822459abc"
+MODEL_PATH = MODEL_DIR / MODEL_FILENAME
+MODEL_CHECKSUM = "d4f763474f9be35ce5358637c87935ee11ba396b9196f73b95ab97da0cef8749"
 
 
 class timer:
@@ -237,3 +242,55 @@ def granular_similarity(metadata_a, metadata_b, threshold=80):
                 matches.append(best_match)
 
     return matches
+
+
+def char_to_byte_offsets(text, char_positions):
+    # type: (str, list[int]) -> list[int]
+    """
+    Efficiently convert character positions to byte positions in a single pass.
+
+    :param text: The input text
+    :param char_positions: List of character positions to convert
+    :return: List of corresponding byte positions
+    """
+    if not char_positions:
+        return []
+
+    # Sort positions for efficient single-pass processing
+    sorted_positions = sorted(set(char_positions))
+    pos_map = {pos: idx for idx, pos in enumerate(sorted_positions)}
+    byte_positions = [0] * len(sorted_positions)
+
+    char_pos = byte_pos = 0
+    pos_idx = 0
+
+    for ch in text:
+        if pos_idx < len(sorted_positions) and char_pos == sorted_positions[pos_idx]:
+            byte_positions[pos_idx] = byte_pos
+            pos_idx += 1
+
+        # Efficient branch-free UTF-8 byte length calculation
+        cp = ord(ch)
+        byte_pos += 1 + (cp >= 0x80) + (cp >= 0x800) + (cp >= 0x10000)
+        char_pos += 1
+
+    # After processing all characters, handle any requested position equal to len(text)
+    while pos_idx < len(sorted_positions) and sorted_positions[pos_idx] == char_pos:
+        byte_positions[pos_idx] = byte_pos
+        pos_idx += 1
+
+    # Map back to original order
+    return [byte_positions[pos_map[pos]] for pos in char_positions]
+
+
+def char_to_byte_offsets_simple(text, char_positions):
+    # type: (str, list[int]) -> list[int]
+    """
+    Simple implementation to convert character positions to byte offsets in a UTF-8 encoded string.
+    This function repeatedly encodes text slices, so its performance is not optimal for large texts.
+
+    :param text: The input text.
+    :param char_positions: List of character positions.
+    :return: List of corresponding byte positions.
+    """
+    return [len(text[:pos].encode("utf-8")) for pos in char_positions]

pyproject.toml

@@ -1,25 +1,28 @@
-[tool.poetry]
+[project]
 name = "iscc-sct"
-version = "0.1.3"
+version = "0.2.2"
 description = "ISCC - Semantic Code Text"
-authors = ["Titusz <tp@py7.de>"]
-license = "CC-BY-NC-SA-4.0"
+authors = [{ name = "Titusz", email = "tp@piscc.io" }]
+requires-python = ">=3.11"
 readme = "README.md"
-homepage = "https://iscc.codes"
-repository = "https://github.com/iscc/iscc-sct"
-documentation = "https://github.com/iscc/iscc-sct"
-keywords=["iscc", "text similarity", "cross lingual", "semantic similarity"]
-classifiers=[
+license = "Apache-2.0"
+license-files = ["LICENSE"]
+keywords = [
+    "iscc",
+    "text similarity",
+    "cross lingual",
+    "semantic similarity",
+]
+classifiers = [
     "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "Intended Audience :: Science/Research",
-    "License :: OSI Approved :: Apache Software License",
     "Natural Language :: English",
     "Operating System :: OS Independent",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Topic :: Text Processing",
     "Topic :: Text Processing :: General",
     "Topic :: Text Processing :: Indexing",
@@ -30,75 +33,117 @@ classifiers=[
     "Topic :: Scientific/Engineering :: Information Analysis",
     "Topic :: Software Development :: Libraries :: Python Modules",
     "Topic :: Software Development :: Libraries",
-    "Topic :: Software Development :: Libraries :: Python Modules",
     "Topic :: System :: Archiving",
     "Topic :: System :: Clustering",
     "Topic :: System :: Distributed Computing",
 ]
+dependencies = [
+    # >=0.32.0 ships the upstream "avoid sizing whole distant split sections" fix and releases
+    # the GIL during native chunking. It mitigates - but does not eliminate - the issue #24
+    # super-linear cost on separator-free spans, so the chunking guard below is retained.
+    "semantic-text-splitter>=0.32.0",
+    "loguru",
+    "blake3",
+    "platformdirs",
+    "tokenizers",
+    "pydantic-settings",
+    "charset-normalizer",
+    "numpy",
+    "pybase64",
+    "certifi",
+    "pyyaml",
+    "pydantic",
+]
 
-[tool.poetry.urls]
-"Changelog" = "https://github.com/iscc/iscc-sct/blob/main/CHANGELOG.md"
-"Bug Tracker" = "https://github.com/iscc/iscc-sct/issues"
-"Twitter" = "https://twitter.com/iscc_foundation"
-"Donate" = "https://iscc.foundation/support"
+# The onnxruntime variant wheels unpack into the same site-packages directory and clobber each
+# other, so exactly one of the cpu/gpu extras must be installed (see issue #23).
+[project.optional-dependencies]
+cpu = ["onnxruntime"]
+gpu = ["onnxruntime-gpu"]
+demo = ["gradio"]
 
-[tool.poetry.scripts]
-sct = 'iscc_sct.cli:main'
+[project.urls]
+Homepage = "https://iscc.codes"
+Repository = "https://github.com/iscc/iscc-sct"
+Documentation = "https://github.com/iscc/iscc-sct"
+Changelog = "https://github.com/iscc/iscc-sct/blob/main/CHANGELOG.md"
+"Bug Tracker" = "https://github.com/iscc/iscc-sct/issues"
+Twitter = "https://twitter.com/iscc_foundation"
+Donate = "https://iscc.foundation/support"
 
-[tool.poetry.dependencies]
-python = ">=3.9,<3.13"
-semantic-text-splitter = "*"
-onnxruntime = "*"
-onnxruntime-gpu = { version = "*", optional = true }
-loguru = "*"
-blake3 = "*"
-platformdirs = "*"
-tokenizers = "*"
-pydantic-settings = "*"
-charset-normalizer = "*"
-numpy = "<2.0.0"
-pybase64 = "*"
-certifi = ">=2024.07.04"
-gradio = { version = "*", optional = true }
-pyyaml = "*"
-pydantic = "*"
+[project.scripts]
+iscc-sct = "iscc_sct.cli:main"
 
+[dependency-groups]
+test = [
+    "pytest",
+    "coverage",
+    "pytest-cov",
+    "onnxruntime",  # dev/test environments use CPU inference
+]
+dev = [
+    "poethepoet",
+    "ruff",
+    "mdformat-mkdocs[recommended]",
+    "mdformat-gfm-alerts",
+    "prek>=0.4.4",
+    "zensical>=0.0.21",
+    "mkdocstrings-python>=1.16.0",
+]
 
-[tool.poetry.extras]
-gpu = ["onnxruntime-gpu"]
-demo = ["gradio"]
+[tool.uv]
+default-groups = "all"
+conflicts = [
+    [{ extra = "cpu" }, { extra = "gpu" }],
+    [{ group = "test" }, { extra = "gpu" }],
+]
 
-[tool.poetry.group.test.dependencies]
-pytest = "*"
-coverage = "*"
-pytest-cov = "*"
+[tool.uv.build-backend]
+module-root = ""
 
-[tool.poetry.group.dev.dependencies]
-poethepoet = "*"
-ruff = "*"
-mdformat-gfm = "*"
-mdformat-gfm-alerts = "*"
-mdformat-frontmatter = "*"
+[build-system]
+requires = ["uv_build>=0.11.20,<0.12.0"]
+build-backend = "uv_build"
 
 [tool.ruff]
 line-length = 100
 
+[tool.ruff.lint]
+# F401: imports referenced only in PEP 484 type comments are invisible to ruff
+ignore = ["F401"]
+
+[tool.ruff.lint.per-file-ignores]
+# F403: star imports intentionally re-export the public API
+"iscc_sct/__init__.py" = ["F403"]
+
 [tool.ruff.format]
 line-ending = "lf"
 
 [tool.coverage.run]
 omit = ["iscc_sct/dev.py", "tests/", "iscc_sct/demo.py"]
+patch = ["subprocess"]
 
 [tool.poe.tasks]
 format-code = { cmd = "ruff format", help = "Code style formating with ruff" }
-format-markdown = { cmd = "mdformat --wrap 100 --end-of-line lf README.md", help = "Markdown formating with mdformat" }
+format-markdown = { cmd = "mdformat --wrap 100 --end-of-line lf --exclude docs/reference/api.md README.md CHANGELOG.md docs", help = "Markdown formating with mdformat" }
 format-yml = { script = "iscc_sct.dev:format_yml", help = "Format YML files"}
 convert-lf = { script = "iscc_sct.dev:convert_lf", help = "Convert line endings to LF"}
-test = { cmd = "pytest --cov=iscc_sct --cov-fail-under=100", help = "Run tests with coverage" }
-update-dependencies = { cmd = "poetry update", help = "Update dependencies" }
-all = ["format-code", "format-markdown", "format-yml", "convert-lf", "test"]
+# Erase any stale .coverage.* parallel data files left by an aborted run before measuring, so the
+# 100% gate never inherits orphaned subprocess data (patch=["subprocess"]) and report a spurious
+# <100% (issue: leftover .coverage.GOLEM.pid* files + failing coverage gate).
+test.sequence = [
+    { cmd = "coverage erase" },
+    { cmd = "pytest --cov=iscc_sct --cov-fail-under=100" },
+]
+test.help = "Run tests with coverage"
+docs-serve = { cmd = "zensical serve", help = "Serve the documentation site locally" }
+docs-build.sequence = [
+    { cmd = "zensical build" },
+    { cmd = "python scripts/gen_llms_full.py" },
+]
+docs-build.help = "Build the docs site and generate llms-full.txt"
+precommit = { cmd = "uv run prek run", help = "Run prek hooks on changed files" }
+export-requirements = { cmd = "uv export --no-default-groups --extra cpu --no-emit-project --no-hashes --no-annotate --no-header --output-file requirements.txt", help = "Export requirements.txt from uv.lock for HF Space" }
+update-dependencies = { cmd = "uv sync --upgrade --extra demo", help = "Update dependencies" }
+all = ["format-code", "format-markdown", "format-yml", "convert-lf", "export-requirements", "test"]
 update = ["update-dependencies", "all"]
-
-[build-system]
-requires = ["poetry-core>=1.0.0"]
-build-backend = "poetry.core.masonry.api"

requirements.txt

@@ -1,11 +1,35 @@
-semantic-text-splitter
-onnxruntime
-loguru
-blake3
-platformdirs
-tokenizers
-pydantic-settings
-charset-normalizer
-numpy<2.0.0
-pybase64
-certifi
+annotated-types==0.8.0
+anyio==4.14.2
+blake3==1.0.9
+certifi==2026.7.22
+charset-normalizer==3.4.9
+click==8.4.2
+colorama==0.4.6 ; sys_platform == 'win32'
+filelock==3.32.2
+flatbuffers==25.12.19
+fsspec==2026.7.0
+h11==0.16.0
+hf-xet==1.5.2 ; platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'
+httpcore==1.0.9
+httpx==0.28.1
+huggingface-hub==1.26.0
+idna==3.18
+loguru==0.7.3
+numpy==2.4.6 ; python_full_version < '3.12'
+numpy==2.5.1 ; python_full_version >= '3.12'
+onnxruntime==1.28.0
+packaging==26.2
+platformdirs==4.11.0
+protobuf==7.35.1
+pybase64==1.4.3
+pydantic==2.13.4
+pydantic-core==2.46.4
+pydantic-settings==2.14.2
+python-dotenv==1.2.2
+pyyaml==6.0.3
+semantic-text-splitter==0.32.0
+tokenizers==0.23.1
+tqdm==4.70.0
+typing-extensions==4.16.0
+typing-inspection==0.4.2
+win32-setctime==1.2.0 ; sys_platform == 'win32'

scripts/gen_llms_full.py

@@ -0,0 +1,106 @@
+"""Generate llms-full.txt and per-page .md files for LLM consumption.
+
+Runs after `zensical build` (see the `docs-build` poe task). Copies cleaned Markdown
+(frontmatter and abbreviation snippets stripped) into site/ alongside the rendered HTML,
+and concatenates every page into site/llms-full.txt.
+
+The API reference page (`reference/api.md`) is all mkdocstrings `:::` directives that only
+render via the plugin, so its plain-text export is a pointer to the rendered page and the
+For Coding Agents API map rather than the raw directives.
+"""
+
+import re
+from pathlib import Path
+
+DOCS_DIR = Path(__file__).parent.parent / "docs"
+SITE_DIR = Path(__file__).parent.parent / "site"
+
+# Ordered list of doc pages to include (relative to docs/), matching the zensical.toml nav
+PAGES = [
+    "index.md",
+    "tutorials/getting-started.md",
+    "howto/compare-texts.md",
+    "howto/granular-features.md",
+    "howto/configuration.md",
+    "howto/command-line.md",
+    "explanation/how-it-works.md",
+    "reference/api.md",
+    "reference/for-coding-agents.md",
+]
+
+# Page whose source is mkdocstrings directives; exported as a pointer instead of raw markdown.
+API_PAGE = "reference/api.md"
+API_POINTER = """# API reference
+
+The API reference is generated from source docstrings and rendered with full signatures at
+https://sct.iscc.codes/reference/api/.
+
+For a plain-text API map of every public symbol - its source module and purpose - see the
+For Coding Agents page (in this file and at https://sct.iscc.codes/reference/for-coding-agents/).
+"""
+
+# Regex to strip YAML frontmatter
+FRONTMATTER_RE = re.compile(r"\A---\n.*?\n---\n", re.DOTALL)
+
+# Regex to strip snippet auto-append directives
+SNIPPET_RE = re.compile(r"^\*\[.*?\]:.*$", re.MULTILINE)
+
+
+def strip_frontmatter(content):
+    # type: (str) -> str
+    """Remove YAML frontmatter from markdown content."""
+    return FRONTMATTER_RE.sub("", content)
+
+
+def strip_snippets(content):
+    # type: (str) -> str
+    """Remove abbreviation snippet definitions appended by pymdownx.snippets."""
+    return SNIPPET_RE.sub("", content)
+
+
+def clean_content(content):
+    # type: (str) -> str
+    """Strip frontmatter, snippets, and normalize whitespace."""
+    content = strip_frontmatter(content)
+    content = strip_snippets(content)
+    return content.strip()
+
+
+def page_content(page):
+    # type: (str) -> str
+    """Return the cleaned export markdown for a page, or the pointer for the API page."""
+    if page == API_PAGE:
+        return API_POINTER.strip()
+    return clean_content((DOCS_DIR / page).read_text(encoding="utf-8"))
+
+
+def main():
+    # type: () -> None
+    """Generate llms-full.txt and individual .md files from doc sources."""
+    SITE_DIR.mkdir(parents=True, exist_ok=True)
+    parts = []
+
+    for page in PAGES:
+        if not (DOCS_DIR / page).exists():
+            print(f"Warning: {page} not found, skipping")
+            continue
+        content = page_content(page)
+        if not content:
+            continue
+        parts.append(content)
+
+        # Write individual .md file to site directory
+        md_path = SITE_DIR / page
+        md_path.parent.mkdir(parents=True, exist_ok=True)
+        md_path.write_text(content + "\n", encoding="utf-8", newline="")
+
+    # Write concatenated llms-full.txt
+    output = "\n\n---\n\n".join(parts) + "\n"
+    out_path = SITE_DIR / "llms-full.txt"
+    out_path.write_text(output, encoding="utf-8", newline="")
+    print(f"Generated {out_path} ({len(parts)} pages, {len(output)} bytes)")
+    print(f"Generated {len(parts)} individual .md files in {SITE_DIR}")
+
+
+if __name__ == "__main__":
+    main()

space.yml

@@ -3,7 +3,7 @@ emoji: 🔮
 colorFrom: red
 colorTo: blue
 sdk: gradio
-sdk_version: 4.41.0
+sdk_version: 5.26.0
 pinned: true
 license: CC-BY-NC-SA-4.0
 short_description: Cross Lingual Similarity Preserving Text Simprints

tests/chunking_vectors.json

@@ -0,0 +1,153 @@
+{
+  "en-default": {
+    "options": {},
+    "text_sha256": "e10a7003e5625c3ff9ec7b400b5d244b6a66a88c1d123f2a255fed5eb8aca648",
+    "offsets": "0 277 612 1006 1224 1491 1743 2043 2334 2648 3183 3462 3845 4146 4411 4653 4939 5305 5629 5875 6313 6627 6851 7197 7504 7907 8157 8354 8745 8963 9331 9668 10033 10229 10425 10783 11199 11378 11663",
+    "sizes": "440 396 431 385 440 380 406 477 415 536 280 449 446 442 443 444 451 485 477 439 517 430 468 394 531 448 421 503 376 403 513 477 393 375 555 533 312 455 413",
+    "chunks_sha256": "363f9a4173919b37d74842c9b4ddaefcd966db3ccb360bd7ca06af1ce9ab272e"
+  },
+  "de-default": {
+    "options": {},
+    "text_sha256": "1fcf71ab8716ba3fd44eb0a8d46c9a58833a1cc2839dfa16244fab7e9dd676e6",
+    "offsets": "0 393 889 1162 1441 1807 2162 2427 2828 3205 3530 3941 4314 4696 5048 5415 5804 6041 6351 6704 7065 7475 7786 8086 8485 8860 9221 9500 10033 10417 10748 11244 11669 12033",
+    "sizes": "546 497 474 476 496 437 448 544 494 326 540 567 527 524 518 546 402 453 565 559 554 392 419 577 599 569 516 534 545 552 560 555 563 606",
+    "chunks_sha256": "1c55035883d3172edec4af391785d3cdcebd04408160e5287a3af5e45e6169a7"
+  },
+  "en-trim": {
+    "options": {
+      "trim": true
+    },
+    "text_sha256": "e10a7003e5625c3ff9ec7b400b5d244b6a66a88c1d123f2a255fed5eb8aca648",
+    "offsets": "2 279 614 1008 1226 1493 1745 2045 2336 2653 3185 3465 3847 4148 4413 4655 4941 5307 5631 5880 6315 6629 6853 7199 7506 7909 8159 8356 8747 8965 9333 9670 10035 10230 10427 10785 11201 11380 11665",
+    "sizes": "437 393 428 382 437 377 403 474 410 530 277 445 443 439 440 441 448 482 474 433 514 427 465 391 528 445 418 500 373 400 510 474 390 373 552 530 309 452 410",
+    "chunks_sha256": "b5c0939c79f227eebbca578b7cba7d869a743667b8bc05866ccfb856ab84eceb"
+  },
+  "de-overlap0": {
+    "options": {
+      "overlap": 0
+    },
+    "text_sha256": "1fcf71ab8716ba3fd44eb0a8d46c9a58833a1cc2839dfa16244fab7e9dd676e6",
+    "offsets": "0 546 1066 1572 1994 2515 3039 3531 4070 4634 5220 5679 6206 6777 7314 7836 8354 8912 9468 10034 10578 11032 11513 12036",
+    "sizes": "546 520 506 422 521 524 492 539 564 586 459 527 571 537 522 518 558 556 566 544 454 481 523 603",
+    "chunks_sha256": "5c9f00e67136002b0d66bb1dc5e1e2fa5045d58365e3fc92ce79ab677ad52bac"
+  },
+  "de-small-chunks": {
+    "options": {
+      "max_tokens": 32,
+      "overlap": 16
+    },
+    "text_sha256": "1fcf71ab8716ba3fd44eb0a8d46c9a58833a1cc2839dfa16244fab7e9dd676e6",
+    "offsets": "0 121 266 393 473 545 638 695 758 821 889 984 1063 1104 1152 1209 1286 1362 1441 1519 1638 1713 1807 1881 1994 2059 2162 2241 2309 2427 2514 2607 2690 2828 2970 3039 3105 3205 3322 3392 3458 3531 3603 3668 3755 3829 3941 4069 4201 4230 4314 4401 4482 4507 4633 4696 4805 4908 5048 5163 5219 5317 5415 5497 5565 5678 5805 5875 5960 6041 6080 6206 6275 6351 6494 6548 6674 6736 6776 6915 7013 7103 7262 7314 7369 7475 7619 7680 7786 7867 7954 8024 8086 8205 8272 8353 8485 8633 8699 8784 8912 8986 9083 9221 9323 9428 9500 9576 9737 9820 9903 10034 10103 10247 10331 10417 10502 10652 10748 10860 11032 11126 11244 11386 11513 11590 11669 11753 11881 11941 12036 12113 12206 12231 12384 12440 12584",
+    "sizes": "124 146 128 81 73 94 57 118 129 95 96 121 102 106 137 105 77 80 131 119 147 95 75 113 129 104 82 69 121 88 96 84 139 143 69 147 101 117 137 137 73 138 124 87 148 113 129 133 114 171 171 107 80 127 97 145 149 141 116 57 99 99 111 120 114 127 137 86 123 81 126 127 77 143 142 157 103 100 140 151 91 160 52 125 107 144 140 107 81 154 124 63 119 130 82 133 149 66 160 128 170 141 139 132 145 72 162 161 167 156 131 154 145 85 123 132 151 97 115 172 154 119 143 127 152 80 130 131 61 95 144 119 93 154 85 161 55",
+    "chunks_sha256": "84e6a9a44d117a455236f42e4e59ea61c978d15155cfc8437e4b8e6389e8c9ca"
+  },
+  "en-byte-offsets": {
+    "options": {
+      "byte_offsets": true
+    },
+    "text_sha256": "e10a7003e5625c3ff9ec7b400b5d244b6a66a88c1d123f2a255fed5eb8aca648",
+    "offsets": "0 277 612 1006 1224 1491 1743 2043 2334 2648 3183 3462 3845 4146 4411 4653 4939 5305 5629 5875 6313 6627 6851 7197 7504 7907 8157 8354 8745 8963 9331 9668 10033 10229 10425 10783 11199 11378 11663",
+    "sizes": "440 396 431 385 440 380 406 477 415 536 280 449 446 442 443 444 451 485 477 439 517 430 468 394 531 448 421 503 376 403 513 477 393 375 555 533 312 455 413",
+    "chunks_sha256": "363f9a4173919b37d74842c9b4ddaefcd966db3ccb360bd7ca06af1ce9ab272e"
+  },
+  "en-pathological": {
+    "options": {},
+    "text_sha256": "61a9478a65146e5f1df1aef776a3c5869ace10f1fdb4507a5c85f93e1d311999",
+    "offsets": "0 277 612 1006 1224 1491 1743 2043 2334 2648 3183 3462 3845 4146 4411 4653 4939 5305 5629 5875 6313 6627 6851 7197 7504 7907 8157 8354 8745 8963 9331 9668 10033 10229 10425 10783 11199 11378 11663",
+    "sizes": "440 396 431 385 440 380 406 477 415 536 280 449 446 442 443 444 451 485 477 439 517 430 468 394 531 448 421 503 376 403 513 477 393 375 555 533 312 455 420",
+    "chunks_sha256": "bdf3103cde11b6ef81652e31910e334e461fe6b720028d1960f3ca7ba4d5a596"
+  },
+  "de-pathological": {
+    "options": {},
+    "text_sha256": "55a001750306eaeac5aec38ef431db29fafa93aae3aba5af1f787ccf06b4ae73",
+    "offsets": "0 393 889 1162 1441 1807 2162 2427 2828 3205 3530 3941 4314 4696 5048 5415 5804 6041 6351 6704 7065 7475 7786 8086 8485 8860 9221 9500 10033 10417 10748 11244 11669 12033 12638",
+    "sizes": "546 497 474 476 496 437 448 544 494 326 540 567 527 524 518 546 402 453 565 559 554 392 419 577 599 569 516 534 545 552 560 555 563 608 8",
+    "chunks_sha256": "da6b12e7e9b878088e886a1f77ca8cb9eb6a2875d9df0af98b5bdc04021a46ce"
+  },
+  "de-pathological-trim": {
+    "options": {
+      "trim": true
+    },
+    "text_sha256": "55a001750306eaeac5aec38ef431db29fafa93aae3aba5af1f787ccf06b4ae73",
+    "offsets": "2 395 891 1167 1443 1809 2164 2431 2831 3208 3533 3943 4317 4698 5050 5417 5806 6043 6353 6706 7067 7477 7788 8088 8487 8862 9223 9501 10035 10422 10750 11246 11671 12037 12641",
+    "sizes": "543 494 471 468 493 432 443 539 488 322 536 564 521 521 515 543 397 450 562 556 549 389 416 574 596 566 513 532 542 546 557 552 560 601 5",
+    "chunks_sha256": "a5528fe8d172d8818101215117b4b958611fd1a3dda04525a1e2550711e24b81"
+  },
+  "en-collapsed-no-breaks": {
+    "options": {},
+    "text_sha256": "e10a7003e5625c3ff9ec7b400b5d244b6a66a88c1d123f2a255fed5eb8aca648",
+    "offsets": "0 277 612 1006 1224 1491 1743 2043 2334 2648 3183 3462 3845 4146 4411 4653 4939 5305 5629 5875 6313 6627 6851 7197 7504 7907 8157 8354 8745 8963 9331 9668 10033 10229 10425 10783 11199 11378 11663",
+    "sizes": "440 396 431 385 440 380 406 477 415 536 280 449 446 442 443 444 451 485 477 439 517 430 468 394 531 448 421 503 376 403 513 477 393 375 555 533 312 455 413",
+    "chunks_sha256": "363f9a4173919b37d74842c9b4ddaefcd966db3ccb360bd7ca06af1ce9ab272e"
+  },
+  "en-crlf": {
+    "options": {},
+    "text_sha256": "f7a1aa8aeb6f97fd89cb85aca27dbf778cba13012073111c1f6dcc899971a8e3",
+    "offsets": "0 280 620 1018 1241 1512 1766 2069 2362 2679 3221 3503 3889 4194 4461 4708 4998 5368 5698 5946 6391 6713 6944 7293 7604 8011 8264 8464 8858 9080 9453 9792 10160 10361 10558 10918 11339 11521 11808",
+    "sizes": "447 403 437 394 446 385 411 482 423 544 284 454 452 449 452 451 458 493 486 447 532 440 475 401 537 455 427 510 385 410 517 483 400 378 561 541 317 461 423",
+    "chunks_sha256": "8ddddf9437056739f6856c2b4b14815032fc88b75b32aad94bc9e34b54fa3433"
+  },
+  "de-multi-blank": {
+    "options": {},
+    "text_sha256": "1fcf71ab8716ba3fd44eb0a8d46c9a58833a1cc2839dfa16244fab7e9dd676e6",
+    "offsets": "0 393 889 1162 1441 1807 2162 2427 2828 3205 3530 3941 4314 4696 5048 5415 5804 6041 6351 6704 7065 7475 7786 8086 8485 8860 9221 9500 10033 10417 10748 11244 11669 12033",
+    "sizes": "546 497 474 476 496 437 448 544 494 326 540 567 527 524 518 546 402 453 565 559 554 392 419 577 599 569 516 534 545 552 560 555 563 606",
+    "chunks_sha256": "1c55035883d3172edec4af391785d3cdcebd04408160e5287a3af5e45e6169a7"
+  },
+  "cjk-pathological": {
+    "options": {},
+    "text_sha256": "188538c93773ba43c5c399dfa60460e74dfe69328fc3ebf15799adea2bb4093e",
+    "offsets": "0 132 264 396 528 660 792 924 1056 1188 1320 1452 1584 1716 1848 1980 2112 2244 2376 2508 2640 2772 2904 3036 3168 3300 3432 3564 3696 3828 3960 4092 4224 4356 4488 4620 4752 4884 5016 5148 5280 5412 5544 5676 5808 5940 6072 6204 6336 6468 6600 6732 6864 6996 7128 7260 7392 7524 7656 7788 7920 8052 8184 8316 8448 8580 8712 8844 8976 9108 9240 9372 9504 9636 9768 9900 10032 10164 10296 10428 10560 10692 10824",
+    "sizes": "198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 198 179",
+    "chunks_sha256": "71502f8844d62063ac8b843e59bc4ff98aa8bf543c0ac16304fb5bfccffb714f"
+  },
+  "long-word-pathological": {
+    "options": {},
+    "text_sha256": "e92803ea4acfdda6a33324b409f8425506a12edfc403d42476c44586f8302e9f",
+    "offsets": "0 304 608 912 1216 1520 1824 2128 2432 2736 3040 3344 3648 3952 4256 4560 4864 5168 5472 5776 6080 6384 6688 6992 7296 7600 7904 8208 8512 8816 9120 9424 9728 10032 10336 10640 10944",
+    "sizes": "483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 483 468",
+    "chunks_sha256": "f69abfad67650c9087ecce78e097c3cab72390eefe1c027860f51e0f05ad97c0"
+  },
+  "unk-runs-pathological": {
+    "options": {},
+    "text_sha256": "61ee460db1d82bf3def0e0588b66c20875edecfbf89e4016ba2c429df41698a9",
+    "offsets": "0 3939 7878",
+    "sizes": "6363 6363 4249",
+    "chunks_sha256": "967815a4207392ec55c33d2dda9a79432c79cc75d0111d66a67dd85df8380f16"
+  },
+  "nbsp-pathological": {
+    "options": {},
+    "text_sha256": "a0eef59ac4ac26fb0a8c3125739d2727e72dfb56f966a2a83c6df4f44eef5c50",
+    "offsets": "0 553 1106 1659 2212 2765 3318 3871 4424 4977 5530 6083 6636 7189 7742 8295 8848 9401 9954 10507 11060 11613 12166 12719 13272 13825 14378 14931 15484 16037 16590 17143 17696 18249 18802 19355 19908 20461 21014 21567 22120 22673 23226 23779 24332 24885 25438 25991 26544 27097 27650",
+    "sizes": "889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 889 356",
+    "chunks_sha256": "a31a1e947114f456e7530f8f107d4fb7e76e36d68f3d268c201311e606fe2a6c"
+  },
+  "mixed-level-pathological": {
+    "options": {},
+    "text_sha256": "9ae546991aedeff306af6bb68537de7dd39c60d519497963859130d68d312039",
+    "offsets": "0 452 906 1360 1814 2268 2722 3176 3630 4084 4538 4992 5446 5900 6354 6808 7262 7716 8170 8624 9078 9532 9986 10440 10894 11348 11802 12256 12710 13164 13618 14072 14526 14980 15434 15888 16342 16796 17250 17704 18158 18612 19066 19520 19974 20428 20882 21336 21790 22244",
+    "sizes": "454 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 456 463",
+    "chunks_sha256": "122f9cfc24b1c0a43a29bb411f964204fc6bc218d764aa76da9d3ccec7194841"
+  },
+  "whitespace-only": {
+    "options": {},
+    "text_sha256": "25ffbca1db7cf0e7f6f5e5146829f328b761a5d4dec01a5de1b522f88f309ad4",
+    "offsets": "0",
+    "sizes": "11",
+    "chunks_sha256": "25ffbca1db7cf0e7f6f5e5146829f328b761a5d4dec01a5de1b522f88f309ad4"
+  },
+  "tiny": {
+    "options": {},
+    "text_sha256": "e3b7d16a6eb31eebba3412b159991654121183f95c16de64139f1a762c8e9595",
+    "offsets": "0",
+    "sizes": "41",
+    "chunks_sha256": "e3b7d16a6eb31eebba3412b159991654121183f95c16de64139f1a762c8e9595"
+  },
+  "unicode-mix": {
+    "options": {},
+    "text_sha256": "e83cf7d12f7514899f21be066f858d23c094b27c60ef4ff836553604df5fbd6f",
+    "offsets": "0",
+    "sizes": "34",
+    "chunks_sha256": "e83cf7d12f7514899f21be066f858d23c094b27c60ef4ff836553604df5fbd6f"
+  }
+}

tests/test_chunking_vectors.py

@@ -0,0 +1,150 @@
+# -*- coding: utf-8 -*-
+"""Chunking test vectors that freeze the exact `split_text` boundaries.
+
+Chunk boundaries determine granular simprints and the final Semantic-Code, so any change to
+the chunking pipeline (text-splitter version, tokenizer, chunk sizer implementation) must
+reproduce these vectors bit-exactly to remain backward compatible.
+
+The vectors in `chunking_vectors.json` pin chunk offsets and sizes for a variety of text
+shapes (including the pathological PDF-extraction shape from issue #24) and option
+combinations. Texts are built deterministically from the committed fixtures `en.txt` and
+`de.txt` plus small synthetic literals; a sha256 of each input text guards against drift
+between the vector file and the text construction below.
+
+Regenerate the vector file (ONLY for an intentional, versioned chunking algorithm change):
+
+    uv run python tests/test_chunking_vectors.py
+"""
+
+import hashlib
+import json
+import re
+from pathlib import Path
+
+import pytest
+
+from iscc_sct.code_semantic_text import split_text
+
+
+HERE = Path(__file__).parent.absolute()
+VECTORS_PATH = HERE / "chunking_vectors.json"
+
+
+def collapse_newlines(text):
+    # type: (str) -> str
+    """Collapse all newline runs to single newlines (PDF-extraction-like text shape)."""
+    return re.sub(r"[\r\n]+", "\n", text)
+
+
+def pathological(text):
+    # type: (str) -> str
+    """
+    Apply the issue #24 trigger shape to text.
+
+    No blank lines anywhere except a single trailing paragraph break. This makes the
+    text-splitter probe from each chunk position all the way to the distant trailing
+    separator, which exposed the super-linear chunking behavior.
+    """
+    return collapse_newlines(text) + "\n\nEnde."
+
+
+def vector_cases():
+    # type: () -> dict[str, tuple[str, dict]]
+    """Build all vector cases as {name: (text, split_text options)}."""
+    en = (HERE / "en.txt").read_text(encoding="utf-8")
+    de = (HERE / "de.txt").read_text(encoding="utf-8")
+    cjk = "数据是新的石油它推动着现代经济的发展与变革。" * 500
+    long_word = "hypermodularization" * 600
+    unk_runs = ("𓀀" * 100 + " ") * 120
+    level3 = ("Ein kurzer Absatz über die Dinge des Lebens. " * 5 + "\n\n") * 100 + "\n\nEnde."
+    # PDF-extraction shape whose words are separated by NBSP (U+00A0) instead of ASCII space;
+    # routed to the guarded splitter and exercises its Unicode-whitespace cut search.
+    nbsp = (chr(0xA0).join(["Inhalt"] * 4000)) + "\n\nEnde."
+    return {
+        "en-default": (en, {}),
+        "de-default": (de, {}),
+        "en-trim": (en, {"trim": True}),
+        "de-overlap0": (de, {"overlap": 0}),
+        "de-small-chunks": (de, {"max_tokens": 32, "overlap": 16}),
+        "en-byte-offsets": (en, {"byte_offsets": True}),
+        "en-pathological": (pathological(en), {}),
+        "de-pathological": (pathological(de), {}),
+        "de-pathological-trim": (pathological(de), {"trim": True}),
+        "en-collapsed-no-breaks": (collapse_newlines(en), {}),
+        "en-crlf": (en.replace("\n", "\r\n"), {}),
+        "de-multi-blank": (de.replace("\n\n", "\n\n\n\n"), {}),
+        "cjk-pathological": (cjk + "\n\n完", {}),
+        "long-word-pathological": (long_word + " Ende\n\nEnde.", {}),
+        "unk-runs-pathological": (unk_runs + "\n\nEnde.", {}),
+        "nbsp-pathological": (nbsp, {}),
+        "mixed-level-pathological": (level3, {}),
+        "whitespace-only": ("  \t \n\n     ", {}),
+        "tiny": ("Hello, World! Schöne Grüße aus München. 😀", {}),
+        "unicode-mix": (
+            "Café ‍naïve 😀🎉 سلام z̧álgo\n\nEnde.",
+            {},
+        ),
+    }
+
+
+def text_hash(text):
+    # type: (str) -> str
+    """Return the sha256 hex digest of text encoded as UTF-8."""
+    return hashlib.sha256(text.encode("utf-8")).hexdigest()
+
+
+def chunks_hash(chunks):
+    # type: (list[str]) -> str
+    """Return the sha256 hex digest over all chunk contents."""
+    return hashlib.sha256("\x1f".join(chunks).encode("utf-8")).hexdigest()
+
+
+def load_vectors():
+    # type: () -> dict
+    """Load the frozen chunking vectors from chunking_vectors.json."""
+    with VECTORS_PATH.open(encoding="utf-8") as f:
+        return json.load(f)
+
+
+def build_vectors():
+    # type: () -> dict
+    """Compute chunking vectors for all cases with the current implementation."""
+    vectors = {}
+    for name, (text, options) in vector_cases().items():
+        result = split_text(text, **options)
+        offsets = [offset for offset, _ in result]
+        sizes = [len(chunk) for _, chunk in result]
+        vectors[name] = {
+            "options": options,
+            "text_sha256": text_hash(text),
+            "offsets": " ".join(str(o) for o in offsets),
+            "sizes": " ".join(str(s) for s in sizes),
+            "chunks_sha256": chunks_hash([chunk for _, chunk in result]),
+        }
+    return vectors
+
+
+def test_vector_file_covers_all_cases():
+    assert set(load_vectors()) == set(vector_cases())
+
+
+@pytest.mark.parametrize("name", vector_cases())
+def test_chunking_vector(name):
+    text, options = vector_cases()[name]
+    expected = load_vectors()[name]
+    assert text_hash(text) == expected["text_sha256"], "input text construction drifted"
+    result = split_text(text, **options)
+    offsets = " ".join(str(offset) for offset, _ in result)
+    sizes = " ".join(str(len(chunk)) for _, chunk in result)
+    assert offsets == expected["offsets"]
+    assert sizes == expected["sizes"]
+    assert chunks_hash([chunk for _, chunk in result]) == expected["chunks_sha256"]
+
+
+if __name__ == "__main__":  # pragma: no cover
+    VECTORS_PATH.write_text(
+        json.dumps(build_vectors(), indent=2, ensure_ascii=False) + "\n",
+        encoding="utf-8",
+        newline="\n",
+    )
+    print(f"Wrote {len(build_vectors())} vectors to {VECTORS_PATH}")

tests/test_cli.py

@@ -1,8 +1,11 @@
 import subprocess
+import sys
 import pytest
-import shutil
 
-sct = shutil.which("sct")
+
+def get_sct_command():
+    """Get the command to run the SCT CLI"""
+    return [sys.executable, "-m", "iscc_sct.cli"]
 
 
 @pytest.fixture
@@ -27,39 +30,55 @@ def non_utf8_text_file(tmp_path):
 
 
 def test_cli_no_args():
-    result = subprocess.run([sct], capture_output=True, text=True)
+    result = subprocess.run(get_sct_command(), capture_output=True, text=True)
     assert result.returncode == 0
     assert "Generate Semantic" in result.stdout
 
 
 def test_cli_empty_file(empty_text_file):
-    result = subprocess.run([sct, str(empty_text_file), "-d"], capture_output=True, text=True)
+    result = subprocess.run(
+        [*get_sct_command(), str(empty_text_file), "-d"], capture_output=True, text=True
+    )
     assert result.returncode == 0
     assert "SKIPPED" in result.stderr
 
 
 def test_cli_non_utf8_file(non_utf8_text_file):
-    result = subprocess.run([sct, str(non_utf8_text_file), "-d"], capture_output=True, text=True)
+    result = subprocess.run(
+        [*get_sct_command(), str(non_utf8_text_file), "-d"], capture_output=True, text=True
+    )
     assert result.returncode == 0
     assert "Could not decode" in result.stderr
     assert "ISCC:" in result.stdout
 
 
 def test_cli_generate_sct(sample_text_file):
-    result = subprocess.run([sct, str(sample_text_file)], capture_output=True, text=True)
+    result = subprocess.run(
+        [*get_sct_command(), str(sample_text_file)], capture_output=True, text=True
+    )
     assert result.returncode == 0
     assert "ISCC:" in result.stdout
 
 
 def test_cli_generate_sct_granular(sample_text_file):
     result = subprocess.run(
-        [sct, str(sample_text_file), "--granular"], capture_output=True, text=True
+        [*get_sct_command(), str(sample_text_file), "--granular"], capture_output=True, text=True
     )
     assert result.returncode == 0
     assert "features" in result.stdout
 
 
 def test_cli_debug_mode(sample_text_file):
-    result = subprocess.run([sct, str(sample_text_file), "--debug"], capture_output=True, text=True)
+    result = subprocess.run(
+        [*get_sct_command(), str(sample_text_file), "--debug"], capture_output=True, text=True
+    )
     assert result.returncode == 0
     assert "DEBUG" in result.stderr
+
+
+def test_cli_doctor():
+    # The test environment has a working CPU runtime, so doctor reports OK and exits cleanly
+    # without prompting (no stdin, no pip call).
+    result = subprocess.run([*get_sct_command(), "doctor"], capture_output=True, text=True)
+    assert result.returncode == 0
+    assert "ONNX runtime" in result.stdout

tests/test_doctor.py

@@ -0,0 +1,246 @@
+"""Tests for the iscc-sct doctor command (ONNX runtime diagnostics and opt-in repair)."""
+
+import builtins
+
+from iscc_sct import doctor
+
+
+def _report(providers, gpu_dist=False, gpu_present=False, cpu_dist=False):
+    """Build a diagnosis report from raw facts."""
+    return doctor._diagnose(providers, gpu_dist, gpu_present, cpu_dist)
+
+
+# --- detection helpers ---
+
+
+def test_dist_installed_true():
+    # onnxruntime is present in the test environment via the `test` dependency group
+    assert doctor._dist_installed("onnxruntime") is True
+
+
+def test_dist_installed_false():
+    assert doctor._dist_installed("definitely-not-a-real-distribution-xyz") is False
+
+
+def test_onnx_providers_returns_list():
+    providers = doctor._onnx_providers()
+    assert isinstance(providers, list)
+    assert "CPUExecutionProvider" in providers
+
+
+def test_onnx_providers_missing(monkeypatch):
+    def _raise(name):
+        raise ImportError(name)
+
+    monkeypatch.setattr(doctor.importlib, "import_module", _raise)
+    assert doctor._onnx_providers() is None
+
+
+def test_gpu_present_returns_bool():
+    assert isinstance(doctor._gpu_present(), bool)
+
+
+def test_doctor_report_returns_dict():
+    report = doctor.doctor_report()
+    assert report["status"] in {"ok", "missing", "shadowed"}
+
+
+# --- pure diagnosis ---
+
+
+def test_diagnose_missing_cpu():
+    report = _report(None)
+    assert report["status"] == "missing"
+    assert report["extra"] == "cpu"
+    assert report["fix"] == [["install", "iscc-sct[cpu]"]]
+    assert report["hint"] is False
+
+
+def test_diagnose_missing_gpu():
+    report = _report(None, gpu_present=True)
+    assert report["status"] == "missing"
+    assert report["extra"] == "gpu"
+    assert report["fix"] == [["install", "iscc-sct[gpu]"]]
+
+
+def test_diagnose_shadowed():
+    # Genuine clobber: BOTH onnxruntime and onnxruntime-gpu installed, CUDA disabled.
+    report = _report(["CPUExecutionProvider"], gpu_dist=True, gpu_present=True, cpu_dist=True)
+    assert report["status"] == "shadowed"
+    assert report["fix"][0][0] == "uninstall"
+    assert report["fix"][1] == ["install", "--force-reinstall", "iscc-sct[gpu]"]
+
+
+def test_diagnose_no_cuda_cpu_only_host():
+    # onnxruntime-gpu correctly installed, no CUDA, nothing shadowing it (CPU-only host).
+    # Not "shadowed": reinstalling cannot help an environment problem, so no fix is offered.
+    report = _report(["CPUExecutionProvider"], gpu_dist=True, gpu_present=False)
+    assert report["status"] == "no_cuda"
+    assert report["fix"] == []
+
+
+def test_diagnose_no_cuda_with_gpu_present():
+    # NVIDIA GPU detected but CUDA unavailable (driver/library issue), gpu package not shadowed.
+    report = _report(["CPUExecutionProvider"], gpu_dist=True, gpu_present=True)
+    assert report["status"] == "no_cuda"
+    assert report["fix"] == []
+
+
+def test_diagnose_broken_gpu_import_is_missing():
+    # onnxruntime-gpu installed but un-importable (providers=None) is "missing", not "shadowed".
+    report = _report(None, gpu_dist=True)
+    assert report["status"] == "missing"
+    assert report["fix"] == [["install", "iscc-sct[cpu]"]]
+
+
+def test_diagnose_ok_cuda():
+    report = _report(["CUDAExecutionProvider", "CPUExecutionProvider"], gpu_dist=True)
+    assert report["status"] == "ok"
+    assert report["cuda"] is True
+    assert report["hint"] is False
+
+
+def test_diagnose_ok_cpu_no_gpu():
+    report = _report(["CPUExecutionProvider"])
+    assert report["status"] == "ok"
+    assert report["hint"] is False
+
+
+def test_diagnose_ok_hint_for_gpu_without_runtime():
+    report = _report(["CPUExecutionProvider"], gpu_present=True)
+    assert report["status"] == "ok"
+    assert report["hint"] is True
+
+
+# --- formatting ---
+
+
+def test_format_report_ok():
+    text = doctor.format_report(_report(["CPUExecutionProvider"]))
+    assert "OK" in text
+    assert "onnxruntime" in text
+
+
+def test_format_report_hint():
+    text = doctor.format_report(_report(["CPUExecutionProvider"], gpu_present=True))
+    assert "CUDA acceleration" in text
+
+
+def test_format_report_missing():
+    text = doctor.format_report(_report(None))
+    assert "no ONNX runtime" in text
+    assert "not installed" in text
+    assert "iscc-sct[cpu]" in text
+
+
+def test_format_report_shadowed():
+    text = doctor.format_report(
+        _report(["CPUExecutionProvider"], gpu_dist=True, gpu_present=True, cpu_dist=True)
+    )
+    assert "shadowed" in text
+    assert "onnxruntime-gpu" in text
+    assert "conflicting" in text  # _runtime_label names both installed packages
+
+
+def test_format_report_no_cuda():
+    text = doctor.format_report(_report(["CPUExecutionProvider"], gpu_dist=True))
+    assert "CUDA is unavailable" in text
+    assert "environment issue" in text
+    assert "onnxruntime-gpu" in text
+
+
+# --- confirmation ---
+
+
+def test_confirm_assume_yes():
+    assert doctor._confirm(True) is True
+
+
+def test_confirm_yes(monkeypatch):
+    monkeypatch.setattr(builtins, "input", lambda prompt="": "Y")
+    assert doctor._confirm(False) is True
+
+
+def test_confirm_no(monkeypatch):
+    monkeypatch.setattr(builtins, "input", lambda prompt="": "")
+    assert doctor._confirm(False) is False
+
+
+def test_confirm_eof(monkeypatch):
+    def _raise(prompt=""):
+        raise EOFError
+
+    monkeypatch.setattr(builtins, "input", _raise)
+    assert doctor._confirm(False) is False
+
+
+# --- run_doctor orchestration ---
+
+
+class _Completed:
+    def __init__(self, returncode):
+        self.returncode = returncode
+
+
+def _fake_runner(calls, returncode=0):
+    def _run(cmd, *args, **kwargs):
+        calls.append(cmd)
+        return _Completed(returncode)
+
+    return _run
+
+
+def test_run_doctor_ok(monkeypatch):
+    monkeypatch.setattr(doctor, "doctor_report", lambda: _report(["CPUExecutionProvider"]))
+    calls = []
+    monkeypatch.setattr(doctor.subprocess, "run", _fake_runner(calls))
+    assert doctor.run_doctor() == 0
+    assert calls == []  # no pip call for a healthy runtime
+
+
+def test_run_doctor_missing_assume_yes(monkeypatch):
+    monkeypatch.setattr(doctor, "doctor_report", lambda: _report(None))
+    calls = []
+    monkeypatch.setattr(doctor.subprocess, "run", _fake_runner(calls, returncode=0))
+    assert doctor.run_doctor(assume_yes=True) == 0
+    assert len(calls) == 1
+    assert calls[0][-1] == "iscc-sct[cpu]"
+
+
+def test_run_doctor_decline(monkeypatch):
+    monkeypatch.setattr(doctor, "doctor_report", lambda: _report(None))
+    monkeypatch.setattr(builtins, "input", lambda prompt="": "n")
+    calls = []
+    monkeypatch.setattr(doctor.subprocess, "run", _fake_runner(calls))
+    assert doctor.run_doctor() == 1
+    assert calls == []
+
+
+def test_run_doctor_shadowed_runs_two_commands(monkeypatch):
+    monkeypatch.setattr(
+        doctor,
+        "doctor_report",
+        lambda: _report(["CPUExecutionProvider"], gpu_dist=True, cpu_dist=True),
+    )
+    calls = []
+    monkeypatch.setattr(doctor.subprocess, "run", _fake_runner(calls, returncode=0))
+    assert doctor.run_doctor(assume_yes=True) == 0
+    assert len(calls) == 2
+
+
+def test_run_doctor_no_cuda(monkeypatch):
+    # no_cuda has no actionable fix: print and exit 0 without prompting or pip calls.
+    monkeypatch.setattr(
+        doctor, "doctor_report", lambda: _report(["CPUExecutionProvider"], gpu_dist=True)
+    )
+    calls = []
+    monkeypatch.setattr(doctor.subprocess, "run", _fake_runner(calls))
+    assert doctor.run_doctor() == 0
+    assert calls == []
+
+
+def test_run_doctor_fix_failure(monkeypatch):
+    monkeypatch.setattr(doctor, "doctor_report", lambda: _report(None))
+    calls = []
+    monkeypatch.setattr(doctor.subprocess, "run", _fake_runner(calls, returncode=1))
+    assert doctor.run_doctor(assume_yes=True) == 1

tests/test_gen_llms_full.py

@@ -0,0 +1,49 @@
+"""Tests for the docs llms-full.txt generation helpers in scripts/gen_llms_full.py."""
+
+import importlib.util
+from pathlib import Path
+
+SCRIPT = Path(__file__).parent.parent / "scripts" / "gen_llms_full.py"
+
+
+def _load_module():
+    # type: () -> object
+    """Load the gen_llms_full script as a module by file path."""
+    spec = importlib.util.spec_from_file_location("gen_llms_full", SCRIPT)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+gen = _load_module()
+
+
+def test_clean_content_strips_frontmatter_and_snippets():
+    content = (
+        "---\nicon: lucide/code\ndescription: x\n---\n"
+        "# Title\n\n"
+        "Body paragraph.\n\n"
+        "*[ISCC]: International Standard Content Code\n"
+    )
+    result = gen.clean_content(content)
+    assert result.startswith("# Title")
+    assert "icon:" not in result
+    assert "*[ISCC]:" not in result
+    assert "Body paragraph." in result
+
+
+def test_api_page_exports_pointer_not_directives():
+    # The API page is mkdocstrings directives; its export must be the pointer, never raw `:::`.
+    assert gen.API_PAGE in gen.PAGES
+    pointer = gen.page_content(gen.API_PAGE)
+    assert ":::" not in pointer
+    assert "reference/api/" in pointer
+    assert "reference/for-coding-agents/" in pointer
+
+
+def test_page_content_cleans_normal_page():
+    page = "explanation/how-it-works.md"
+    src = gen.DOCS_DIR / page
+    result = gen.page_content(page)
+    assert result == gen.clean_content(src.read_text(encoding="utf-8"))
+    assert not result.startswith("---")

tests/test_iscc_sct.py

@@ -1,9 +1,11 @@
+from importlib.metadata import version
 from pathlib import Path
 
 import pytest
 from blake3 import blake3
 
 import iscc_sct as sct
+from iscc_sct import code_semantic_text as cst
 from iscc_sct.code_semantic_text import (
     split_text,
     tokenize_chunks,
@@ -31,7 +33,8 @@ be matched based on lexical similarity.
 
 
 def test_version():
-    assert sct.__version__ == "0.1.3"
+    # Compared against the package metadata so the two cannot drift apart silently
+    assert sct.__version__ == version("iscc-sct")
 
 
 def test_code_text_semantic_default():
@@ -104,44 +107,42 @@ def test_gen_text_code_semantic_granular():
         offsets=True,
         contents=True,
     )
-    assert (
-        result
-        == {
-            "characters": 726,
-            "iscc": "ISCC:CAARISHPJHEXQAYL",
-            "features": [
-                {
-                    "maintype": "semantic",
-                    "subtype": "text",
-                    "version": 0,
-                    "simprints": ["FWjtTcl4Aws", "lAjHSc1wAws"],
-                    "offsets": [0, 297],
-                    "contents": [
-                        "\n"
-                        "`iscc-sct` is a **proof of concept implementation** of a semantic "
-                        "Text-Code for the\n"
-                        "[ISCC](https://core.iscc.codes) (*International Standard Content "
-                        "Code*). Semantic Text-Codes are\n"
-                        "designed to capture and represent the language agnostic semantic "
-                        "content of text for improved\n"
-                        "similarity detection.\n"
-                        "\n",  # NOTE: end of first chunk (see comma :)
-                        "\n"
-                        "\n"
-                        "The ISCC framework already comes with a Text-Code that is based "
-                        "on lexical similarity and can match\n"
-                        "near duplicates. The ISCC Semantic Text-Code is planned as a new "
-                        "additional ISCC-UNIT focused on\n"
-                        "capturing a more abstract and broad semantic similarity. As such "
-                        "the Semantic Text-Code is\n"
-                        "engineered to be robust against a broader range of variations and "
-                        "translations of text that cannot\n"
-                        "be matched based on lexical similarity.\n",
-                    ],
-                }
-            ],
-        }
-    )
+    assert result == {
+        "characters": 726,
+        "iscc": "ISCC:CAARISHPJHEXQAYL",
+        "features": [
+            {
+                "maintype": "semantic",
+                "subtype": "text",
+                "version": 0,
+                "byte_offsets": False,
+                "simprints": ["FWjtTcl4Aws", "lAjHSc1wAws"],
+                "offsets": [0, 297],
+                "contents": [
+                    "\n"
+                    "`iscc-sct` is a **proof of concept implementation** of a semantic "
+                    "Text-Code for the\n"
+                    "[ISCC](https://core.iscc.codes) (*International Standard Content "
+                    "Code*). Semantic Text-Codes are\n"
+                    "designed to capture and represent the language agnostic semantic "
+                    "content of text for improved\n"
+                    "similarity detection.\n"
+                    "\n",  # NOTE: end of first chunk (see comma :)
+                    "\n"
+                    "\n"
+                    "The ISCC framework already comes with a Text-Code that is based "
+                    "on lexical similarity and can match\n"
+                    "near duplicates. The ISCC Semantic Text-Code is planned as a new "
+                    "additional ISCC-UNIT focused on\n"
+                    "capturing a more abstract and broad semantic similarity. As such "
+                    "the Semantic Text-Code is\n"
+                    "engineered to be robust against a broader range of variations and "
+                    "translations of text that cannot\n"
+                    "be matched based on lexical similarity.\n",
+                ],
+            }
+        ],
+    }
 
 
 def test_gen_text_code_semantic_checks_bits():
@@ -156,15 +157,107 @@ def test_split_text(text_en):
 
 
 def test_split_text_override():
-    text = "Try some very small and granular text splitting. Use options override for it."
+    text = "Try some very small and granular text splitting with Iñtërnâtiônàlizætiøn☃. Use options override for it."
     chunks = split_text(text, max_tokens=8, overlap=4)
     assert chunks == [
         (0, "Try some very small and granular text "),
-        (20, "and granular text splitting. "),
-        (49, "Use options override for it."),
+        (20, "and granular text splitting with "),
+        (53, "Iñtërnâtiônà"),
+        (59, "âtiônàlizætiøn"),
+        (73, "☃. "),
+        (76, "Use options override for it."),
+    ]
+
+
+def test_split_text_override_byte_offsets():
+    text = "Try some very small and granular text splitting with Iñtërnâtiônàlizætiøn☃. Use options override for it."
+    chunks = split_text(text, max_tokens=8, overlap=4, byte_offsets=True)
+    assert chunks == [
+        (0, "Try some very small and granular text "),
+        (20, "and granular text splitting with "),
+        (53, "Iñtërnâtiônà"),
+        (61, "âtiônàlizætiøn"),
+        (80, "☃. "),
+        (85, "Use options override for it."),
     ]
 
 
+def test_split_text_resource_options_do_not_fragment_splitter_cache():
+    # The splitter cache is keyed on its kwargs - resource knobs like batch_size must not
+    # reach it, or every distinct value would retain another tokenizer-backed TextSplitter.
+    split_text("Hello World", batch_size=11)
+    cached = cst.splitter.cache_info().currsize
+    split_text("Hello World", batch_size=12, intra_op_threads=3)
+    assert cst.splitter.cache_info().currsize == cached
+
+
+def test_chunking_tokenizer_disables_truncation_and_padding():
+    # The sizing tokenizer must not truncate/pad: truncation makes the HF chunk sizer emit
+    # overflow encodings and degrades chunking to super-linear runtime (issue #24). The
+    # embedding tokenizer must keep truncation so ISCC codes stay unchanged.
+    chunking = cst.chunking_tokenizer()
+    assert chunking.truncation is None
+    assert chunking.padding is None
+    assert cst.tokenizer().truncation is not None
+
+
+def test_token_count_matches_guarded_for_small_text():
+    text = "Ein kurzer deutscher Satz mit Umlauten: äöüß."
+    assert cst.token_count_guarded(text, 127) == cst.token_count(text)
+
+
+def test_token_count_guarded_short_circuits_oversized_text():
+    text = "Viele kleine Wörter ergeben einen sehr langen Text. " * 100
+    guarded = cst.token_count_guarded(text, 127)
+    assert guarded > 127
+    assert cst.token_count(text) > 127
+
+
+def test_token_count_guarded_falls_back_on_sparse_prefix():
+    # Prefix probe stays below max_tokens, forcing exact full tokenization
+    text = "a" + " " * 2000 + "b" * 2000
+    assert cst.token_count_guarded(text, 127) == cst.token_count(text)
+
+
+def test_token_count_guarded_short_circuits_unicode_whitespace():
+    # NBSP-separated words are tokenizer word boundaries (WhitespaceSplit), so the guard must
+    # short-circuit on them instead of falling back to full tokenization (issue #24 PDF whitespace).
+    text = chr(0xA0).join(["wort"] * 4000)  # NBSP-separated words
+    guarded = cst.token_count_guarded(text, 127)
+    # The overestimate path returns prefix_count + tail-chars, strictly above the exact count;
+    # a fallback would instead return exactly token_count(text).
+    assert guarded > cst.token_count(text)
+
+
+def test_needs_split_guard_paragraph_gap():
+    # Large gap from the start to the next paragraph separator routes to the guarded splitter.
+    assert cst.needs_split_guard("x" * 20000 + "\n\nEnde.") is True
+
+
+def test_needs_split_guard_newline_free_span():
+    # A giant span with no newline at all (one long paragraph) also blows up the native sizer.
+    assert cst.needs_split_guard("word " * 4000) is True
+
+
+def test_needs_split_guard_long_line_before_newline():
+    # A >8K separator-free span before the first single newline is enough to route to guarded.
+    assert cst.needs_split_guard("word " * 4000 + "\nrest") is True
+
+
+def test_needs_split_guard_trailing_separator_free_span():
+    # Early paragraph break then a long separator-free tail (missed by the paragraph-level scan).
+    assert cst.needs_split_guard("Intro.\n\n" + "word " * 4000) is True
+
+
+def test_needs_split_guard_dense_newlines_is_false():
+    # Dense single newlines bound the probes; no oversized span -> normal splitter.
+    assert cst.needs_split_guard("line\n" * 4000) is False
+
+
+def test_needs_split_guard_short_text_is_false():
+    assert cst.needs_split_guard("A short paragraph.\n\nAnother one.") is False
+
+
 def test_tokenize_chunks():
     chunks = ["Hello World", "These are chunks"]
     result = tokenize_chunks(chunks)
@@ -190,6 +283,73 @@ def test_embed_chunks():
     assert list(embeddings[0][:3]) == pytest.approx(expected, rel=1e-3)
 
 
+def test_embed_chunks_batching_is_result_invariant():
+    # Batch size is a pure resource knob - it must not change the embeddings. Chunks differ in
+    # length so the batched run pads, which is exactly what could perturb the result.
+    chunks = ["Hello World", "These are chunks", "A considerably longer chunk of sample text"]
+    assert embed_chunks(chunks, batch_size=1) == pytest.approx(
+        embed_chunks(chunks, batch_size=3), abs=1e-6
+    )
+
+
+def test_embed_chunks_default_uses_global_batch_size(monkeypatch):
+    # soft_hash_text_semantic calls embed_chunks without a batch size - the documented global
+    # override (ISCC_SCT_BATCH_SIZE) must still apply on that path.
+    calls = []
+    real_tokenize = cst.tokenize_chunks
+    monkeypatch.setattr(cst, "tokenize_chunks", lambda c: calls.append(len(c)) or real_tokenize(c))
+    monkeypatch.setattr(sct.sct_opts, "batch_size", 2)
+    embed_chunks(["Hello World", "These are chunks", "A third chunk"])
+    assert calls == [2, 1]
+
+
+def test_resolve_batch_size_explicit_wins():
+    assert cst.resolve_batch_size(7, ["CPUExecutionProvider"]) == 7
+    assert cst.resolve_batch_size(7, ["CUDAExecutionProvider"]) == 7
+
+
+def test_resolve_batch_size_auto():
+    assert cst.resolve_batch_size(0, ["CPUExecutionProvider"]) == cst.CPU_BATCH_SIZE
+    assert (
+        cst.resolve_batch_size(0, ["CUDAExecutionProvider", "CPUExecutionProvider"])
+        == cst.GPU_BATCH_SIZE
+    )
+
+
+def test_session_options_thread_default():
+    rt = cst.load_onnxruntime()
+    # 0 leaves the runtime default in place instead of pinning a thread count
+    assert cst.session_options(rt, 0).intra_op_num_threads == 0
+    assert cst.session_options(rt, 2).intra_op_num_threads == 2
+
+
+def capture_gpu_shadowed_warnings(providers):
+    # type: (list[str]) -> list[str]
+    """Run warn_gpu_shadowed with a capturing log sink and return emitted warnings."""
+    messages = []
+    handler_id = cst.log.add(messages.append, level="WARNING")
+    try:
+        cst.warn_gpu_shadowed(providers)
+    finally:
+        cst.log.remove(handler_id)
+    return messages
+
+
+def test_warn_gpu_shadowed_silent_with_cuda():
+    assert capture_gpu_shadowed_warnings(["CUDAExecutionProvider", "CPUExecutionProvider"]) == []
+
+
+def test_warn_gpu_shadowed_silent_without_gpu_package():
+    assert capture_gpu_shadowed_warnings(["CPUExecutionProvider"]) == []
+
+
+def test_warn_gpu_shadowed_warns_on_shadowed_install(monkeypatch):
+    monkeypatch.setattr(cst, "distribution", lambda name: object())
+    messages = capture_gpu_shadowed_warnings(["CPUExecutionProvider"])
+    assert len(messages) == 1
+    assert "CUDA support is unavailable" in messages[0]
+
+
 def test_gen_text_code_semantic(text_en):
     result = sct.gen_text_code_semantic(text_en, embedding=True)
     assert result["iscc"] == "ISCC:CAA636IXQD736IGJ"
@@ -242,6 +402,99 @@ def test_compress():
     assert compress(arr2, 1) == expected
 
 
+def test_utf32be_chunk_retrieval():
+    """Test that we can retrieve text chunks using UTF-32BE encoding with offset/size * 4."""
+    # Generate text code with features
+    text = (
+        "Hello world! 你好世界! こんにちは! 안녕하세요! مرحبا! שלום! Ç 가 Ω ℍ ① ︷ i⁹ ¼ dž ⫝̸ ȴ ȷ ɂ ć "
+        "Iñtërnâtiôn\nàlizætiøn☃💩 –  is a tric\t ky   thing!\r"
+    )
+    text += TEXT
+    result = sct.gen_text_code_semantic(
+        text, simprints=True, offsets=True, sizes=True, contents=True
+    )
+
+    # Convert text to UTF-32BE
+    text_utf32be = text.encode("utf-32be")
+
+    # For each feature, retrieve the chunk using offset and size
+    features = result["features"][0]
+    for i, simprint in enumerate(features["simprints"]):
+        offset = features["offsets"][i]
+        size = features["sizes"][i]
+        original_chunk = features["contents"][i]
+
+        # Calculate byte offset and size in UTF-32BE
+        byte_offset = offset * 4
+        byte_size = size * 4
+
+        # Retrieve chunk from UTF-32BE encoded text
+        chunk_bytes = text_utf32be[byte_offset : byte_offset + byte_size]
+        retrieved_chunk = chunk_bytes.decode("utf-32be")
+
+        # Verify retrieved chunk matches the original
+        assert retrieved_chunk == original_chunk, f"Chunk mismatch at index {i}"
+
+
 def test_embedding_precision():
     d16 = sct.gen_text_code_semantic("Hello World", embedding=True, precision=4)
     assert d16["features"][0]["embedding"][0] == 0.0087
+
+
+def test_create_byte_offsets():
+    """Test generation with byte offsets using text with multibyte characters."""
+    # Text with ASCII, CJK, emoji, etc.
+    text = (
+        "Hello world! 你好世界! こんにちは! 안녕하세요! مرحبا! שלום! Ç 가 Ω ℍ ① ︷ i⁹ ¼ dž ⫝̸ ȴ ȷ ɂ ć "
+        "Iñtërnâtiôn\nàlizætiøn☃💩 –  is a tric\t ky   thing!\r"
+    )
+    text += TEXT
+
+    # Generate with character offsets (default)
+    result_char = sct.create(text, offsets=True, sizes=True)
+    # Generate with byte offsets
+    result_bytes = sct.create(text, offsets=True, sizes=True, byte_offsets=True)
+
+    # ISCC and Character Count should be the same
+    assert result_char.iscc == result_bytes.iscc
+    assert result_char.characters == result_bytes.characters
+
+    # Features should be different
+    assert result_char.features != result_bytes.features
+
+    # Character versus UTF-8 chunk retrieval and actual chunks should match
+    char_chunk = text[0 : result_char.features[0].sizes[0]]
+    byte_chunk = text.encode("utf-8")[0 : result_bytes.features[0].sizes[0]].decode("utf-8")
+    assert char_chunk == byte_chunk
+    actual_chunk = sct.create(text, contents=True)
+    assert actual_chunk.features[0].contents[0] == char_chunk
+
+    assert result_char.model_dump(exclude_none=True) == {
+        "iscc": "ISCC:CAARISGPJHEXQBQL",
+        "characters": 851,
+        "features": [
+            {
+                "byte_offsets": False,
+                "maintype": "semantic",
+                "offsets": [0, 209, 422],
+                "sizes": [307, 215, 429],
+                "subtype": "text",
+                "version": 0,
+            }
+        ],
+    }
+
+    assert result_bytes.model_dump(exclude_none=True) == {
+        "iscc": "ISCC:CAARISGPJHEXQBQL",
+        "characters": 851,
+        "features": [
+            {
+                "byte_offsets": True,
+                "maintype": "semantic",
+                "offsets": [0, 280, 493],
+                "sizes": [378, 215, 429],
+                "subtype": "text",
+                "version": 0,
+            }
+        ],
+    }

tests/test_main.py

@@ -21,6 +21,7 @@ def test_create_granular():
                 "maintype": "semantic",
                 "subtype": "text",
                 "version": 0,
+                "byte_offsets": False,
                 "simprints": [
                     {"content": "Hello World", "offset": 0, "simprint": "82eJ2NG741E", "size": 11}
                 ],

tests/test_models.py

@@ -26,6 +26,7 @@ def test_feature_set_initialization():
         "maintype": "semantic",
         "subtype": "text",
         "version": 0,
+        "byte_offsets": False,
     }