File size: 7,370 Bytes

70ba45f

---
license: apache-2.0
language:
- es
tags:
- psychology
- clinical
- spanish
- sft
- axolotl
base_model:
- google/gemma-4-E4B-it
---
# ALIA-es-gemma-clinical-psychology-sft

This repository contains a supervised fine-tuned (SFT) version of the **Gemma 4 E4B IT** model, optimized for Spanish psychological counseling and empathetic therapeutic dialogue.

This model is the result of a **Supervised Fine-Tuning (SFT)** process on the [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) base model, using a curated multi-turn psychology dataset in Spanish containing professional therapist dialogues.

> [!NOTE]
> This is a **pilot training run** for research purposes. It is not an official release and has not been validated for general deployment.

> [!WARNING]
> **DISCLAIMER:** This model is a domain-specific proof-of-concept for therapeutic guidance and research. It has *NOT* been clinically validated and has not undergone regulatory review.
> It may produce incorrect, unsafe, or misleading psychological advice. Do not use this model as a substitute for professional therapy, diagnosis, or psychiatric treatment. Always consult a qualified psychologist or healthcare professional.

---

## Model Details

### Description
This model is a Transformer-based decoder-only language model that builds on the **Gemma 4 E4B IT** architecture through SFT alignment targeted to psychological support in Spanish.

**SFT Fine-Tuning:** The model was fine-tuned using Supervised Fine-Tuning (SFT) to align responses with empathetic, active listening strategies. The dataset consists of multi-turn dialogues between patients and therapists, reinforcing safe, validating, and explorative conversational practices.

### Architecture
|                         |               |
|-------------------------|:--------------|
| **Base Model**          | google/gemma-4-E4B-it |
| **Architecture**        | Dense + PLE (Parameter-Layer-Embedding) |
| **Total Parameters**    | 8,000,000,000 (8B) |
| **Effective Parameters**| 4,500,000,000 (4.5B) |
| **Layers**              | 42            |
| **Shared KV cache layers**| 18           |
| **Context length**      | 4,096 (Configured) |
| **Attention Pattern**   | Alternating local sliding-window (512 tokens) and global full-context |
| **Precision**           | bfloat16      |
| **Flash attention**     | ❌ (Disabled) |

### Hyperparameters
| Parameter | Value |
|---|---:|
| **Sequence length** | 4,096 |
| **Sample packing** | false |
| **Pad to sequence length** | true |
| **Num. epochs** | 20 |
| **Save steps** | 40 |
| **Eval steps** | 20 |
| **Logging steps** | 5 |
| **Optimizer** | adamw_torch |
| **Learning rate** | 5e-5 |
| **LR scheduler** | cosine |
| **Warmup ratio** | 0.05 |
| **Weight decay** | 0.01 |
| **Micro batch size** | 1 |
| **Gradient accumulation steps** | 2 |
| **Gradient checkpointing** | true |
| **Val set size** | 0.05 |
| **Seed** | 42 |
| **BF16** | true |
| **FP16** | false |
| **Fine-Tuning Method** | SFT (Supervised Fine-Tuning) |
| **Adapter** | LoRA |
| **LoRA R** | 16 |
| **LoRA Alpha** | 32 |
| **LoRA Dropout** | 0.05 |
| **LoRA Target Modules** | `model.language_model.layers.[\d]+.(_checkpoint_wrapped_module.)?(mlp|self_attn).(up|down|gate|q|k|v|o)_proj` |

---

## Intended Use

### Direct Use
The model is intended for research, development and support applications within **Spanish psychological counseling and active listening** contexts. Representative use cases include:
- Assisting mental health professionals with drafts or suggestions for empathetic response strategies.
- Role-playing and scenario training for psychology students.
- Analysis and study of automated fine-tuning in clinical-adjacent communication settings.

### Out-of-scope Use
This model is not approved for clinical use or autonomous deployment. It must not be used as a primary source for psychological diagnosis, psychiatric treatment decisions, or crisis intervention. Any deployment that impacts patient safety requires extensive validation, risk assessment and regulatory clearance.

---

## How to use

### Python Example

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "SINAI/ALIA-es-gemma-clinical-psychology-sft"

# System prompt used to steer the model towards empathetic therapy
system_prompt = (
    "Eres un terapeuta psicológico empático y profesional. "
    "Escucha activamente al paciente y responde de forma apropiada, "
    "validando sus emociones y explorando su experiencia."
)

# Example conversation structure (using the Gemma 4 chat template style)
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Hola. Pues llevo desde hace mucho sintiendome con mucha ansiedad por basicamente casi todo"}
]

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

# Apply the chat template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)

# Decode the generated tokens (skipping the prompt part)
generated_tokens = outputs[0][inputs.input_ids.shape[1]:]
print(tokenizer.decode(generated_tokens, skip_special_tokens=True))
```

---

## Data

### SFT Fine-Tuning Data
> [!NOTE]
> **Data Availability:** The dataset used for training is publicly available on Hugging Face at [SINAI/ALIA-es-clinical-psychology-dialogues](https://huggingface.co/datasets/SINAI/ALIA-es-clinical-psychology-dialogues).

To adapt the model to empathetic psychological counseling in Spanish we used the following resource:

* **Supervised Fine-Tuning (SFT)**
    * **Dataset:** `ALIA-es-clinical-psychology-dialogues.jsonl` (containing 67 multi-turn sessions)
    * **Description:** A multi-turn dataset structured with a `conversations` history. The target responses (`assistant`) are human-curated/edited therapist dialogues that prioritize validation, empathy, active listening, and safe, explorative therapeutic interaction in Spanish.

---

## Additional Information

### License
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)

### Citation
```bibtex
@misc{ALIA-es-gemma-clinical-psychology-sft,
    title={ALIA-es-gemma-clinical-psychology-sft: Empathetic Psychology SFT Model for Spanish},
    author={SINAI Research Group},
    year={2026},
    publisher={HuggingFace},
    howpublished={\url{https://huggingface.co/SINAI/ALIA-es-gemma-clinical-psychology-sft}}
}
```

Please also cite the base model and family:

```bibtex
@misc{gemma4_2026,
    title={Gemma 4: Open Weights Multimodal Models},
    author={Google DeepMind},
    year={2026},
    url={https://deepmind.google/gemma}
}
```

### Funding
This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project [ALIA](https://alia.gob.es).

---

**Contact:** [ALIA Project](https://www.alia.gob.es/) - [SINAI Research Group](https://sinai.ujaen.es) - [Universidad de Jaén](https://www.ujaen.es/)

**More Information:** [SINAI Research Group](https://sinai.ujaen.es) | [ALIA-UJA Project](https://github.com/sinai-uja)