---
license: mit
datasets:
- OpenAssistant/oasst1
- HuggingFaceFW/fineweb
language:
- en
pipeline_tag: text-generation
tags:
- conversational
- generative
- fast
- efficient
- great
- tasks
- agent
- gpt
- text-generation-inference
- art
library_name: transformers
base_model:
- Aobangaming/lightning-30m-ft
- Aobangaming/lightning-60m
---

# Model Card for Model ID

## Model Details

We introduce LUNA, a small, autoregressive transformer. This model aims to provide conversational-like chat without
overloading the computer. This model is designed to run on small hardware, such as phones or office computers.

- Model creator: [AobanZ](https://aobanweb.com)

### Model Description

Luna utilizes FlashAttention and AdamW for performance and capability. Lightning is designed to provide quick, coherent outputs, improved with
a larger size and weight.
- **Developed by:** AobanZ
- **Model type:** Transformer
- **Language(s) (NLP):** English
- **License:** MIT

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** https://huggingface.co/Aobangaming/luna-1.5-flash

## Uses

Luna is intended to be used for research, analysis and fine-tuning, general conversation, stories and other.
It is not intended to be used for professional advice, real writing or any kind of heavy work as
generated outputs may be incorrect.

### Direct Use

Luna can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities.

Direct use is primarily intended for research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.


### Downstream Use

Luna may be fined-tuned for a AI Character, AI Agents, and chat models. However, please note that
generated outputs may be corrupted and/or incorrect.


### Out-of-Scope Use

Heavy Work may overload the model, which will cause corrupted outputs and/or misinformation if implemented
into a larger-app/ecosystem.

## Bias, Risks, and Limitations

Lightning is designed to process english and conversational text ONLY and cannot be fined-tuned for any other uses(eg. Robotics)

### Recommendations

We recommend users of Lightning to finetune the model on new text, 
and add necessary guardrails and precautions to prevent misuse.

## How to Get Started with the Model

Use the code below to get started with the model.
```python
import torch
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
import importlib.util

model_id = "Aobangaming/luna-1.5-flash"

path = hf_hub_download(model_id, "modeling_lightning.py")
spec = importlib.util.spec_from_file_location("lightning", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True
)
tokenizer = Tokenizer.from_file(
    hf_hub_download(model_id, "luna_tokenizer.json")
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

history = []

print("Aoban Luna 1.5 Flash")
print("!clear = clear history | !exit = quit")

while True:
    prompt = input("You: ").strip()

    if prompt.lower() == "!exit":
        break

    if prompt.lower() == "!clear":
        history.clear()
        print("History cleared.")
        continue

    response = module.generate_text(
        model,
        tokenizer,
        prompt,
        max_len=100,
        device=device,
        top_k=40,
        top_p=0.6,
        penalty=1.2,
        temperature=0.8,
        chat_history=history
    )

    print(f"Assistant: {response}")

    history.extend([
        {"role": "user", "content": prompt},
        {"role": "assistant", "content": response}
    ])
```

## Training Details

### Training Data

Luna was trained a subset of the [OASST](https://huggingface.co/datasets/OpenAssistant/oasst1) dataset.

### Training Procedure

Luna was trained on an RTX 3050 GPU, using FlashAttention/SDPA and MHA.
The model was trained on a large dataset. It was not trained on fine-tuning datasets since memory issues.

## Training Results
| Epoch |        Loss | Perplexity |
| ----: | ----------: | ---------: |
|     1 |     6.13648 |     462.42 |
|     2 |     4.84533 |     127.15 |
|     3 |     4.17732 |      65.19 |
|     4 |     3.67652 |      39.51 |
|     5 | **3.27694** |  **26.49** |
#### Training Hyperparameters

| Hyperparameter | Value | Comment |
| :--- | :--- | :--- |
| Precision | FP32 | 
| Optimizer | AdamW | 
| Learning rate | 5e-4 | 
| Batch size | 32 | 



## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** RTX 3050 6GB
- **Hours used:** 3
- **Cloud Provider:** AobanLabs
- **Compute Region:** Asia
- **Carbon Emitted:** ~0.17 kg CO₂e

## Technical Specifications

### Model Architecture and Objective

Luna uses a 6-layer causal Transformer with 256-dimensional hidden states and 4 attention heads. Each attention head has a dimension of 64.

The architecture uses pre-layer normalization, causal scaled dot-product attention, a 4× expansion GELU feed-forward network, sinusoidal positional encoding, and untied input/output embeddings.

| Hyperparameter | Value | Comment |
| :--- | :--- | :--- |
| Layers | 6 | 
| D_MODEL | 256 | Optimized for 64dim/head
| Attention Heads | 4 | 
| Vocabulary | ~75003 |  w/ 200 Sequence length

### Benchmarks
Aoban Luna 1.5 got a 15% benchmark in a custom-made benchmark generated by AI.

### Compute Infrastructure


#### Hardware

The model was trained on a RTX 3050 6GB paired with a UHD Graphics 630.