---
license: apache-2.0
datasets:
- Allanatrix/Scientific_Research_Tokenized
language:
- en
base_model:
- tiiuae/Falcon3-10B-Base
pipeline_tag: text-generation
tags:
- chemistry
- biology
- physics
---

# Nexa_Sci_distilled_Falcon-10B

LoRA-tuned Falcon3‑10B for high-fidelity scientific question answering and methodology drafting.

---

## Model Overview

|                      | Details                                                                 |
|----------------------|-------------------------------------------------------------------------|
| Base model           | `tiiuae/Falcon3-10B-Base`                                                |
| Method               | QLoRA (4‑bit quantization; rank 64 adapters)                           |
| Trainable parameters | 26,214,400 (≈0.25 % of the base model)                                  |
| Training corpus      | `sft_scientific_v1` (filtered teacher generations across biology, physics, materials science) |
| Training window      | 2025‑11‑05 → 2025‑11‑06                                                 |

### Highlights
- Validation loss: **0.410**
- Test loss: **0.413**
- Generates structured, technically grounded scientific responses
- Rubric (judge) scoring is **pending** for the final checkpoint

---

## Intended Use

- Scientific question answering assistants  
- Lab protocol / methodology drafting  
- Hypothesis ideation and critique

Not suitable for medical, legal, or other high-stakes decisions without human oversight.

---

## Quickstart

### 1. Load merged weights (single-step deployment)

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "NexaCompute/Nexa_Sci_distilled_Falcon-10B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "Explain how superconductors enable near-lossless power transmission in urban grids."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.2,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### 2. Load as LoRA adapter (if you prefer to manage the base model separately)

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "tiiuae/Falcon3-10B-Base",
    device_map="auto",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)

adapter = PeftModel.from_pretrained(base, "NexaCompute/Nexa_Sci_distilled_Falcon-10B")
adapter = adapter.merge_and_unload()  # optional
tokenizer = AutoTokenizer.from_pretrained("NexaCompute/Nexa_Sci_distilled_Falcon-10B")
```

---

## Sample Output

**Prompt**  
*Design a reproducible lab experiment to measure photosynthesis rate in freshwater algae.*

**Model output (abridged)**  
```
1. Hypothesis: CO₂ enrichment increases dissolved O₂ production in Chlorella vulgaris.
2. Materials: 3× 500 mL photobioreactors, dissolved oxygen probes, LED array (250 µmol·m⁻²·s⁻¹), CO₂ regulator...
3. Procedure:
   - Equilibrate cultures at 25 °C for 48 h.
   - Pressurize headspace to 0.5, 1.0, 1.5 atm CO₂ for 40 min intervals.
   - Record dissolved O₂ every 5 min; compute linear slopes over 30 min windows.
4. Analysis: Fit O₂ flux vs CO₂ concentration; report slope, R², and p-value.
```

---

## Training & Infrastructure

### Data
Filtered teacher generations across biology, physics, and materials science. Only examples passing dual-judge quality gates were retained (`sft_scientific_v1`).

### Hyperparameters
- Effective batch size: 32 (QLoRA, gradient accumulation 4, 2 GPUs)  
- Epochs: 3 (early stop at step 200)  
- Optimizer: AdamW (lr 2e‑4, weight decay 0.01)  
- Precision: BF16

### Hardware & Cost
- 2 × A100 80 GB (Prime Intellect)  for ~$2/hr
- Estimated training cost ≈ USD 18

---

## Evaluation Summary

| Metric    | Value | Notes                    |
|-----------|-------|--------------------------|
| Val loss  | 0.410 | QLoRA adapters           |
| Test loss | 0.413 | Held-out scientific QA   |
| Judge     | —     | Rubric evaluation pending |

Final rubric scoring will be published once inference benchmarking completes.

---

## Limitations & Risks

- Knowledge cutoff from `Falcon3-10B-Base`; recent findings may be missing.  
- May produce invented citations or experimental details—verify before use.  
- No RLHF/safety fine-tuning; human review is essential.  
- Current PyTorch wheels lack CUDA `sm_120` kernels, so RTX 5090 inference requires rebuilt binaries or alternative hardware.

---

## Responsible Use

- Keep a human in the loop for experimental or safety-critical decisions.  
- Do not deploy in clinical, legal, or security domains without additional validation.  
- Report issues or unsafe behaviors via NexaCompute support channels.

---

## Changelog

- **2025-11-06** — Initial release containing merged weights, tokenizer, and training summary.