Llama-2-7b-HiCI-100k

🔥 A Llama-2-7B with Hierarchical Context Integration (HiCI)

🎯 Key Highlights

• Extended Context: 8,192 tokens (4× base Llama-2)
• Novel Architecture: Hierarchical Context Integration (HiCI)
• 512M Trainable Parameters: Including 380M for HiCI modules
• Based on LongLoRA: Efficient training with shifted sparse attention

🆚 Differences from Standard LongLoRA

Component	Standard LongLoRA	This Model (HiCI)
LoRA Adapters	q,k,v,o projections	✅ Same (27MB)
Embeddings	Token embeddings	✅ Same (500MB)
Normalization	LayerNorm weights	✅ Same (1MB)
🔥 Local Construction	❌ Not included	✅ 269M params (1GB)
🔥 Global Integration	❌ Not included	✅ 111M params (424MB)
Total Size	~1.1 GB	~2 GB

Trade-off: 2× larger checkpoint for hierarchical context conditioning.

Architecture Details

HiCI Module Breakdown

Technical Overview:

Standard self-attention has $\mathcal{O}(T^2)$ complexity. HiCI uses segmented attention with structured context conditioning:

1. Local Construction: Cross-attention with $M$ learnable query slots extracts compact local representations $L_i \in \mathbb{R}^{M imes d}$ from each segment
2. Global Integration: Local representations are aggregated into shared global context $G \in \mathbb{R}^{K imes d}$ via multi-view statistical pooling
3. Top-down Broadcast: Global context $G$ and local abstraction $L_i$ are prepended to each segment's key-value sequence

This achieves $\mathcal{O}(T \cdot S)$ complexity while maintaining cross-segment information flow.

File Structure

adapter_model.bin (27 MB)
├── LoRA Adapters: 13.9M parameters
└── Target: q_proj, k_proj, v_proj, o_proj

trainable_params.bin (2 GB)
├── Local Construction Modules: 269M parameters (1.0 GB) 🔥
├── Global Integration Modules: 111M parameters (424 MB) 🔥
├── Embeddings: 131M parameters (500 MB)
└── Normalization: 0.27M parameters (1 MB)

💻 Usage

Installation

pip install transformers peft torch

Load Model

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load HICI adapter
model = PeftModel.from_pretrained(
    base_model,
    "ZengXiangyu/Llama-2-7b-HiCI-100k"
)

# Load tokenizer  
tokenizer = AutoTokenizer.from_pretrained("ZengXiangyu/Llama-2-7b-HiCI-100k")

# Generate
prompt = "Your long context (up to 16K tokens)..."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚠️ Important Note

This model uses HiCI (Hierarchical Context Integration) modules for enhanced long-context capabilities. The implementation code will be available on GitHub. For now, you can load and use the model with standard PEFT interface as shown above.

## License

Llama 2 Community License

## 🙏 Acknowledgements

- Based on [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b-hf) by Meta AI
- Training methodology from [LongLoRA](https://github.com/dvlab-research/LongLoRA)
- HiCI architecture: Hierarchical Context Integration with segmented attention