File size: 2,024 Bytes
461e98d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
datasets:
- ray0rf1re/FineWeb-Nano
- Nix-ai/Cat-v2.8
- databricks/databricks-dolly-15k
- allenai/ai2_arc
- lighteval/MATH-Hard
- tatsu-lab/alpaca
- HuggingFaceTB/smoltalk
- openai/gsm8k
language:
- en
license: other
license_name: hyper-v2
tags:
- ai-evaluation
- mathematics
- trigonometry
- calculus
- algebra
- qwen2
- causal-lm
- trained-from-scratch
base_model: none
---
# HyperNix.2
**Version:** 0.2
**Parameters:** 101,370,880 (~101.37 M)
**Architecture:** Qwen 2.5-style decoder-only transformer (trained **from scratch**)
## What can it do?
| Domain | Capability |
|---|---|
| AI Evaluation | Test, evaluate, grade and rate other AI model outputs |
| Mathematics | Full Trigonometry, Calculus (limits/derivatives/integrals), Algebra 1 & 2 |
| English | Fluent conversational English |
## Architecture
| Hyperparameter | Value |
|---|---|
| `vocab_size` | 151,936 (Qwen 2.5 tokenizer) |
| `hidden_size` | 512 |
| `intermediate_size` | 1,193 (SwiGLU) |
| `num_hidden_layers` | 9 |
| `num_attention_heads` | 8 |
| `num_key_value_heads` | 4 (GQA) |
| `max_position_embeddings` | 2,048 |
| `tie_word_embeddings` | True |
| Total parameters | **101,370,880** |
## Training
- Hardware: Single NVIDIA GTX 1080 (8 GB VRAM)
- Precision: FP16 + gradient checkpointing
- Optimizer: AdamW (lr=3e-4, cosine decay)
- This is **not** a LoRA or fine-tune — all weights are randomly initialised and trained from scratch.
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/hyper-Nix.2")
model = AutoModelForCausalLM.from_pretrained("ray0rf1re/hyper-Nix.2", torch_dtype=torch.float16)
prompt = "<|im_start|>user\nEvaluate this AI response: The capital of France is London.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(out[0], skip_special_tokens=False))
``` |