---
license: mit
datasets:
- Aobangaming/Conversational-Fine-Tuning
language:
- en
base_model:
- Aobangaming/Aoban-2.7-L
pipeline_tag: text-generation
tags:
- conversational
- generative
---

# Model Card for Model ID

Aoban 3.0 is a large, autoregressive transformer which utilizes FlashAttention and MHA.

## Model Details



### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** AobanZ
- **Model type:** Transformer
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model [optional]:** Aoban-2.7-L

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** https://huggingface.co/Aobangaming/Aoban-3.0-220M

## Uses

Aoban 3.0 is intended to be used for research, analysis and fine-tuning. 
It is not intended to be used for professional advice, or any kind of heavy work as
generated outputs may be incorrect.

### Direct Use

Aoban 3.0 can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities.

Direct use is primarily intended for research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.

### Downstream Use [optional]

Aoban 3.0 may be fined-tuned for an AI Agents, Simple app helpers, and small conversational models. However, please note that
generated outputs may be corrupted and/or incorrect.


### Out-of-Scope Use

Heavy Work may overload the model, which will cause corrupted outputs and/or misinformation if implemented
into a larger-app/ecosystem.

## Bias, Risks, and Limitations

Aoban 3.0 is designed to process english and conversational text ONLY and cannot be fined-tuned for other uses.


### Recommendations

We recommend users of Aoban 3.0 to finetune the model on new text, 
and add necessary guardrails and precautions to prevent misuse.

## How to Get Started with the Model

Use the code below to get started with the model.
The code will be added later. For now use the script 'ai thingy.py' to open an interactive interface in the python console.


## Training Details

### Training Data

Aoban 3.0 was trained on the full [Conversational Fine Tuning](https://huggingface.co/datasets/Aobangaming/Conversational-Fine-Tuning/) dataset.

### Training Procedure

Aoban 3.0 was trained on an RTX 3050 GPU, using FlashAttention and MHA.

#### Preprocessing [optional]

[More Information Needed]


#### Training Hyperparameters

| Hyperparameter | Value | Comment |
| :--- | :--- | :--- |
| Precision | FP32 | |  
| Optimizer | AdamW | | Better weight decay
| Learning rate | 5e-5 | Inherited from Aoban 2.7
| Batch size | 16 |  



## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** RTX 3050 6GB
- **Hours used:** 6
- **Cloud Provider:** My Computer
- **Compute Region:** Asia
- **Carbon Emitted:** 0.21 kg CO₂e

## Technical Specifications [optional]

### Model Architecture and Objective

Aoban 3.0 is a casual decoder model which is autoregressive.

| Hyperparameter | Value | Comment |
| :--- | :--- | :--- |
| Layers | 16 | 
| D_MODEL | 1024 | Optimized for 64dim/head
| Attention Heads | 16 |
| Vocabulary | ~5600 |  w/ 160 Sequence length

### Compute Infrastructure


#### Hardware

RTX 3050 6GB

#### Software

Windows 11, Intel i5-10400