Upload README.md with huggingface_hub

README.md

@@ -1,201 +1,138 @@
 ---
+license: apache-2.0
+license_link: https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/main/LICENSE
+base_model: deepseek-ai/deepseek-moe-16b-base
 tags:
-- pytorch
-- deepseek
+- moe
 - mixture-of-experts
-- text-generation
-- cpu-offloading
 - gguf
-- llama-cpp
-- memory-efficient
-- local-inference
-- moe
+- llama.cpp
+- shimmy
+- rust
+- cpu-offload
+quantized_by: MikeKuykendall
 language:
 - en
-license: other
-model_type: deepseek
-inference: true
+- zh
 pipeline_tag: text-generation
-library_name: transformers
+library_name: llama.cpp
 ---
 
-# DeepSeek MoE 16B with CPU Expert Offloading
+# DeepSeek MoE 16B Base - F16 GGUF with MoE CPU Offloading Support
 
-## Model Description
+F16 GGUF conversion of [deepseek-ai/deepseek-moe-16b-base](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base) with Rust bindings for llama.cpp's MoE CPU offloading functionality.
 
-**DeepSeek MoE 16B CPU Offload** is a memory-optimized GGUF conversion of DeepSeek's MoE 16B model, enhanced with revolutionary CPU expert offloading technology. This enables running a 16.38 billion parameter Mixture of Experts model with minimal GPU memory requirements through innovative expert tensor offloading.
+## Model Details
 
-### Key Features
+- **Base Model**: [deepseek-ai/deepseek-moe-16b-base](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base)
+- **Format**: GGUF F16 precision
+- **File Size**: 31GB
+- **Parameters**: 16.4B total (2.8B active per token)
+- **Architecture**: 28 layers, 64 regular experts + 2 shared experts, 6 active per token
+- **Context Length**: 4K tokens
+- **Converted by**: [MikeKuykendall](https://huggingface.co/MikeKuykendall)
 
-- **🧠 Advanced Architecture**: 64 regular experts + 2 shared experts, 6 active per token
-- **💾 Minimal VRAM Usage**: CPU expert offloading dramatically reduces GPU memory requirements
-- **⚡ Efficient Inference**: Optimized for local deployment with acceptable load times (~40s)
-- **🔧 Production Ready**: Validated working implementation with coherent text generation
-- **📏 Reasonable Context**: 4K token context length for focused tasks
+## MoE CPU Offloading
 
-## Model Specifications
+This model supports **MoE CPU offloading** via llama.cpp (implemented in [PR #15077](https://github.com/ggml-org/llama.cpp/pull/15077)). Shimmy provides Rust bindings for this functionality, enabling:
 
-| Specification | Value |
-|---------------|-------|
-| **Parameters** | 16.38B (total) |
-| **Architecture** | DeepSeek MoE with dual expert system |
-| **Expert Configuration** | 64 regular experts + 2 shared experts |
-| **Active Experts** | 6 per token |
-| **Context Length** | 4,096 tokens |
-| **Precision** | F16 |
-| **File Size** | 32.8GB (GGUF) |
-| **Base Model** | [deepseek-ai/deepseek-moe-16b-base](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base) |
+- **VRAM Reduction**: 92.5% (30.1GB → 2.3GB measured on GH200)
+- **Performance Trade-off**: 4.1x slower generation (26.8 → 6.5 TPS)
+- **Use Case**: Running 16B parameter MoE on consumer GPUs (<4GB VRAM)
 
-## Memory Requirements
+### Controlled Baseline (NVIDIA GH200, N=3)
 
-### Traditional Inference (Estimated)
-- **Full GPU Loading**: ~33-35GB VRAM (based on model size)
-- **CPU RAM**: ~2GB
+| Configuration | VRAM | TPS | TTFT |
+|---------------|------|-----|------|
+| **GPU-only** | 30.1GB | 26.8 | 426ms |
+| **CPU Offload** | 2.3GB | 6.5 | 1,643ms |
 
-### With CPU Expert Offloading ⚡
-- **GPU VRAM**: Minimal (expert tensors offloaded to CPU)
-- **CPU RAM**: ~35GB (includes expert tensors)
-- **Memory Savings**: Significant VRAM reduction while maintaining performance
+**Trade-off**: Memory for speed. Best for VRAM-constrained scenarios where generation speed is less critical than model size.
 
-## Installation & Usage
+### Unique Architecture
 
-### Prerequisites
+DeepSeek MoE uses a **dual-expert architecture** (64 regular + 2 shared experts), validated to work correctly with CPU offloading:
+- Regular experts: `ffn_gate_exps.weight`, `ffn_down_exps.weight`, `ffn_up_exps.weight`
+- Shared experts: `ffn_gate_shexp.weight`, `ffn_down_shexp.weight`, `ffn_up_shexp.weight`
 
-```bash
-# Install required dependencies
-pip install llama-cpp-python
-# OR build llama.cpp with MoE CPU offloading support
-git clone https://github.com/ggerganov/llama.cpp
-cd llama.cpp
-make LLAMA_CUDA=1
-```
-
-### Download Model
+## Download
 
 ```bash
-# Using HuggingFace CLI
 huggingface-cli download MikeKuykendall/deepseek-moe-16b-cpu-offload-gguf \
-  deepseek-moe-16b-f16.gguf --local-dir ./models
+  --include "deepseek-moe-16b-f16.gguf" \
+  --local-dir ./models
 ```
 
-### Basic Usage
+## Usage
 
-```bash
-# Using llama.cpp with CPU expert offloading
-./main -m ./models/deepseek-moe-16b-f16.gguf \
-       --cpu-moe \
-       --prompt "What is mixture of experts in AI?" \
-       --n-predict 100
-```
+### llama.cpp (CPU Offloading)
 
-### Python Integration
-
-```python
-from llama_cpp import Llama
-
-# Initialize model with CPU expert offloading
-llm = Llama(
-    model_path="./models/deepseek-moe-16b-f16.gguf",
-    n_ctx=4096,
-    cpu_moe=True,  # Enable CPU expert offloading
-    verbose=True
-)
+```bash
+# Standard loading (requires ~32GB VRAM)
+./llama-server -m deepseek-moe-16b-f16.gguf -c 4096
 
-# Generate text
-response = llm("What is mixture of experts in AI?", max_tokens=100)
-print(response['choices'][0]['text'])
+# With MoE CPU offloading (requires ~3GB VRAM + 32GB RAM)
+./llama-server -m deepseek-moe-16b-f16.gguf -c 4096 --cpu-moe
 ```
 
-## Performance Benchmarks
-
-### Model Loading
-- **Load Time**: ~40 seconds (including expert tensor initialization)
-- **Memory Initialization**: Expert tensors successfully moved to CPU
-- **Architecture Detection**: 64+2 expert configuration properly recognized
-
-### Generation Quality
-- **Coherence**: Maintains logical flow and context understanding
-- **Technical Accuracy**: Produces contextually appropriate responses
-- **Response Length**: Generates coherent text within token limits
-- **Expert Activation**: All 6 active experts properly utilized
-
-### Memory Efficiency
-- **Expert Tensor Offloading**: ✅ All expert tensors successfully moved to CPU
-- **GPU Memory**: Minimal usage with CPU offloading enabled
-- **Total Model Size**: 32.8GB efficiently distributed between GPU and CPU
+### Shimmy (Rust Bindings)
 
-## Technical Architecture
-
-### Unique Dual Expert System
-DeepSeek MoE implements an innovative architecture combining:
-
-1. **64 Regular Experts**: Standard MoE experts for specialized processing
-2. **2 Shared Experts**: Always-active experts for common patterns
-3. **6 Active Per Token**: 6 experts activated for each token (highest among tested models)
-
-### Expert Tensor Distribution
-```
-Expert Tensors: ffn_gate_exps.weight, ffn_down_exps.weight, ffn_up_exps.weight
-Shared Experts: shared_expert.gate_proj.weight, shared_expert.up_proj.weight, shared_expert.down_proj.weight
-Buffer Override: All expert tensors moved to CPU for memory efficiency
+```bash
+# Install Shimmy
+cargo install --git https://github.com/Michael-A-Kuykendall/shimmy --features llama-cuda
+
+# Standard loading
+shimmy serve --model deepseek-moe-16b-f16.gguf
+
+# With MoE CPU offloading
+shimmy serve --model deepseek-moe-16b-f16.gguf --cpu-moe
+
+# Query the API
+curl http://localhost:11435/api/generate \
+  -d '{
+    "model": "deepseek-moe-16b",
+    "prompt": "Explain the architecture of DeepSeek MoE",
+    "max_tokens": 256,
+    "stream": false
+  }'
 ```
 
-## Comparison with Other MoE Models
-
-| Model | Parameters | Experts | Active/Token | VRAM Reduction | Context |
-|-------|------------|---------|--------------|----------------|---------|
-| **DeepSeek MoE 16B** | 16.38B | 64+2 shared | 6 | High | 4K |
-| GPT-OSS 20B | 20B | 32 | 4 | 99.9% | 131K |
-| Phi-3.5-MoE 41.9B | 41.9B | 16 | 2 | 97.1% | 131K |
+## Performance Notes
 
-## Limitations
+**Standard GPU Loading**:
+- VRAM: 30.1GB
+- Speed: 26.8 TPS
+- Latency: 426ms TTFT
+- Use when: VRAM is plentiful, speed is critical
 
-1. **Context Length**: 4K tokens (shorter than other tested models)
-2. **Generation Patterns**: May exhibit some repetitive patterns requiring parameter tuning
-3. **Expert Complexity**: Dual expert system may require specialized handling for optimal performance
-4. **Load Time**: ~40 second initialization due to large model size and expert configuration
+**CPU Offloading**:
+- VRAM: 2.3GB (92.5% reduction)
+- Speed: 6.5 TPS (4.1x slower)
+- Latency: 1,643ms TTFT
+- Use when: Limited VRAM, speed less critical
 
-## Use Cases
+## Original Model
 
-### Ideal For:
-- **Local AI Development**: Efficient local inference for development and testing
-- **Memory-Constrained Environments**: Systems with limited GPU VRAM but adequate CPU RAM
-- **Research Applications**: Studying MoE architectures and expert activation patterns
-- **Educational Purposes**: Understanding dual expert system architectures
+- **Developers**: DeepSeek AI
+- **License**: Apache 2.0
+- **Paper**: [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://arxiv.org/abs/2401.06066)
+- **Languages**: English, Chinese
 
-### Best Practices:
-- Use with sufficient CPU RAM (>35GB) for optimal performance
-- Consider parameter tuning to reduce repetitive generation patterns
-- Monitor expert activation patterns for insights into model behavior
-- Combine with other models for diverse inference capabilities
+## Technical Validation
 
-## Model Card Authors
-
-**MikeKuykendall** - Conversion, optimization, and CPU offloading implementation
+Full validation report with controlled baselines: [Shimmy MoE CPU Offloading Technical Report](https://github.com/Michael-A-Kuykendall/shimmy/blob/feat/moe-cpu-offload/docs/MOE-TECHNICAL-REPORT.md)
 
 ## Citation
 
-If you use this model in your research, please cite:
-
 ```bibtex
-@misc{deepseek-moe-16b-cpu-offload,
-  title={DeepSeek MoE 16B with CPU Expert Offloading},
-  author={MikeKuykendall},
-  year={2025},
-  url={https://huggingface.co/MikeKuykendall/deepseek-moe-16b-cpu-offload-gguf}
+@article{dai2024deepseekmoe,
+  title={DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models},
+  author={Dai, Damai and others},
+  journal={arXiv preprint arXiv:2401.06066},
+  year={2024}
 }
 ```
 
-## License
-
-This model follows the original DeepSeek license terms. Please refer to the [base model](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base) for complete licensing information.
-
-## Acknowledgments
-
-- **DeepSeek Team**: Original model architecture and training
-- **GGML/llama.cpp Community**: GGUF format and inference optimization
-- **MoE CPU Offloading Research**: Breakthrough memory optimization techniques
-
 ---
 
-*Model converted and optimized as part of comprehensive MoE CPU offloading research - October 2025*
+*GGUF conversion and MoE offloading validation by [MikeKuykendall](https://huggingface.co/MikeKuykendall)*