license: apache-2.0
library_name: peft
tags:
- generated_from_trainer
base_model: BioMistral/BioMistral-7B
model-index:
- name: spanish_medica_llm
results: []
Model Card for SpanishMedicaLLM
Más de 600 millones de personas hablantes del idioma español necesitan recursos, como los LLMs, para la obtención de información médica de forma libre y segura, cumpliendo con los objetivo del milenio: Salud y Bienestar, Educación y Calidad, Fin de la Pobreza propuestos por la ONU. Existen pocos LLM para el dominio médico en idioma español.
El objetivo de este proyecto es crear un gran modelo de lenguaje (LLM; siglas en inglés) para el contexto médico en español permitiendo crear soluciones y servicios de información de salud en LATAM. El modelo contará con información de medicinas convencionales, naturales y tradicionales. Un resultado del proyecto es un conjunto de datos público del dominio médico que agrupa recursos de otras fuentes que permite crear o ajustar LLM. Los resultados del desempeño del LLM se comparan con otros modelos del state-of-the-art como BioMistral, Meditron, MedPalm.
Model Details
Model Description
- Developed by: Dionis López Ramos, Alvaro Garcia Barragan, Dylan Montoya, Daniel Bermúdez
- Funded by: SomosNLP, HuggingFace
- Model type: Language model, instruction tuned
- Language(s): Spanish (
es-ES,es-CL) - License: apache-2.0
- Fine-tuned from model: BioMistral/BioMistral-7B
- Dataset used: somosnlp/SMC/
Model Sources
- Repository: spaces/somosnlp/SpanishMedicaLLM/
- Paper: "Comming soon!"
- Demo: spaces/somosnlp/SpanishMedicaLLM
- Video presentation: SpanishMedicaLLM | Proyecto Hackathon #SomosNLP
Uses
Direct Use
[More Information Needed]
Out-of-Scope Use
Los creadores del LLM no se hacen responsable de resultados nocivos que puedan generar. Se sugiere un proceso de evaluación riguroso con especialistas de los resultados generados.
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
How to Get Started with the Model
Use the code below to get started with the model.
HUB_MODEL_ID = '/somosnlp/spanish_medica_llm'
config = PeftConfig.from_pretrained(HUB_MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(HUB_MODEL_ID, return_dict=True, load_in_8bit=True, device_map={"":0})
Training Details
Training Data
Dataset used was somosnlp/SMC/
Training Procedure
Training Hyperparameters
Training regime:
- learning_rate: 2.5e-05
- train_batch_size: 16
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 5
- training_steps: 2
- mixed_precision_training: Native AMP
Evaluation
Testing Data, Factors & Metrics
Testing Data
Dataset used was a 20% from somosnlp/SMC/
Factors
[More Information Needed]
Metrics
[More Information Needed]
Results
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
- Hardware Type: GPU
- Hours used: 4 Hours
- Cloud Provider: Hugginface
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Model Architecture and Objective
We used a LLM arquitecture to BioMistral/BioMistral-7B because if a foundational model trained in medical domain datasets.
Compute Infrastructure
[More Information Needed]
Hardware
Nvidia T4 Small 4 vCPU 15 GB RAM 16 GB VRAM
Software
- transformers==4.38.0
- torch>=2.1.1+cu113
- trl @ git+https://github.com/huggingface/trl
- peft
- wandb
- accelerate
- datasets
[More Information Needed]
License
Apache License 2.0
Citation
BibTeX:
@software{lopez2024spanishmedicallm,
author = {Lopez Dionis, Garcia Alvaro, Montoya Dylan, Bermúdez Daniel},
title = {SpanishMedicaLLM},
month = February,
year = 2024,
url = {https://huggingface.co/datasets/HuggingFaceTB/cosmopedia}
}
More Information
This project was developed during the Hackathon #Somos600M organized by SomosNLP. The model was trained using GPUs sponsored by HuggingFace.
Team:
Contact
For any doubt or suggestion contact to: PhD Dionis López (inoid2007@gmail.com)