Upload folder using huggingface_hub

README.md

@@ -0,0 +1,87 @@
+---
+license: apache-2.0
+language: [en]
+library_name: transformers
+pipeline_tag: text-generation
+tags: [legal, finance, small-language-model, pretrained-from-scratch]
+---
+
+# slm125m-live
+
+A 125M-parameter Llama-architecture language model **pretrained from scratch** on a
+legal- and finance-heavy corpus. Trained end to end on Modal; the tokenizer is also
+trained from scratch on this corpus (16,384-token byte-level BPE).
+
+This is a **base model**. It has had no instruction tuning, no RLHF, and no safety
+alignment. At 125M parameters it will confabulate freely — it is a research and
+teaching artifact, not a source of legal or financial advice.
+
+## Architecture
+
+| | |
+|---|---|
+| Parameters | 125,847,552 (125.8M) |
+| Layers / hidden / heads | 12 / 768 / 12 (MHA) |
+| Context length | 1024 |
+| Vocab | 16,384 (byte-level BPE, trained on this corpus) |
+| Position encoding | RoPE (theta 10000) |
+| Activation / norm | SwiGLU / RMSNorm |
+| Tied embeddings | yes |
+
+## Training data (2.04B unique tokens)
+
+| Source | Tokens | Share |
+|---|---|---|
+| case-law | 716M | 35.1% |
+| sec | 860M | 42.2% |
+| fineweb-edu | 465M | 22.8% |
+
+Built from `HFforLegal/case-law` (US court opinions), `PleIAs/SEC` (SEC filings) and
+`HuggingFaceFW/fineweb-edu` (`sample-10BT`, general fluency filler). The legal sources
+are the binding constraint: together they hold only ~2B clean tokens, so the mix is
+"take all the legal text, add a small web slice" rather than a chosen ratio.
+
+Pipeline: stream -> 6-step deterministic clean (line filters, boilerplate strip,
+4-gram repetition, ASCII/langdetect English gate, dictionary-based OCR gate on
+case-law) -> MinHash near-dedup + exact dedup -> **13-gram decontamination against
+CaseHOLD/LexGLUE** -> pack into 1024-token windows, 99/1 train/val split.
+
+## Training
+
+| | |
+|---|---|
+| Tokens seen | 8.16B (4 epochs) |
+| Steps | 15,568 |
+| Global batch | 524,288 tokens |
+| Optimizer | AdamW (betas 0.9/0.95, wd 0.1, clip 1.0) |
+| LR schedule | cosine 0.0006 -> 6e-05, 200M warmup tokens |
+| Precision | bf16 autocast, fp32 master weights |
+| Hardware | 8x NVIDIA H100 (DDP, single node) |
+| Final val loss | 2.1228 |
+
+## Evaluation (held-out 1% split)
+
+| Split | Perplexity |
+|---|---|
+| ALL | 8.31 |
+| case-law | 8.68 |
+| sec | 4.80 |
+| fineweb-edu | 21.61 |
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("AnandHaridas1980/slm125m-live")
+model = AutoModelForCausalLM.from_pretrained("AnandHaridas1980/slm125m-live")
+
+ids = tok("The plaintiff filed a motion to dismiss on the grounds that", return_tensors="pt")
+print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=True, top_p=0.95)[0]))
+```
+
+## Limitations
+
+Base model, English only, 1024-token context. The
+case-law source is OCR'd and retains some scanning noise despite the dictionary gate.
+Training data is skewed to older SEC filings. Do not use for legal or financial advice.

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 12,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3",
+  "use_cache": false,
+  "vocab_size": 16384
+}

eval.json

@@ -0,0 +1,20 @@
+{
+  "val_loss": {
+    "ALL": 2.1173862352371216,
+    "case-law": 2.160631491661072,
+    "sec": 1.5677862186431886,
+    "fineweb-edu": 3.073202949523926
+  },
+  "perplexity": {
+    "ALL": 8.309390287501428,
+    "case-law": 8.676615138896363,
+    "sec": 4.7960190952061215,
+    "fineweb-edu": 21.61101091794557
+  },
+  "samples": [
+    "The plaintiff filed a motion to dismiss on the grounds that the action was barred by the statute of limitations and because of the defendant's admission in its answer that the cause of action was barred by the statute of limitations. The court granted the motion and the plaintiff appealed. This court affirmed the judgment of the trial court, holding that the action was barred by the",
+    "IN THE UNITED STATES DISTRICT COURT, the defendant argues that the contract provision was ambiguous and that the court should reform the contract. But the court\u2019s finding that the contract is unambiguous and clear is not clearly erroneous. The court\u2019s finding was supported by the evidence and the court was not clearly erroneous in finding that the defendant\u2019s contract was unambiguous. See State",
+    "Item 7. Management's Discussion and Analysis of Financial Condition. Net revenues of $17.4 million for the year ended December 31, 1995, represent an increase of $10.4 million or 59.4% over 1994 net revenues of $12.6 million. The increase in net revenues from 1994 to 1995 is attributable to increases in the following areas: (i",
+    "The Company's total assets as of December 31 were $1.2 billion and $1.0 billion as of December 31, 1997 and 1996, respectively. During 1997, the Company invested in a portfolio of mortgage backed securities with maturities of approximately 15 years as a means of leveraging its capital, which is the principal source of capital available to invest"
+  ]
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.51.3",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:055378f22367286003c92070f8a36b4e22c43a13bad9cac651c56d6caefef77b
+size 251708904

special_tokens_map.json

@@ -0,0 +1,11 @@
+{
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|system|>"
+  ],
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "pad_token": "<|pad|>",
+  "unk_token": "<|unk|>"
+}

tokenizer_config.json

@@ -0,0 +1,72 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|system|>"
+  ],
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eos|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<|unk|>"
+}

training_summary.json

@@ -0,0 +1,6 @@
+{
+  "steps": 15568,
+  "val_loss": 2.1228176186084746,
+  "tokens_seen": 8162115584,
+  "epochs": 4
+}