Hugging Face's logo

gghfez
/
SmartMaid-123b-GGUF

GGUF
English
conversational
Model card Files
xet
 Community
1
How to use from
llama.cpp
Install (macOS, Linux) 
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
# Run inference directly in the terminal:
llama cli -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
Install from WinGet (Windows) 
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
# Run inference directly in the terminal:
llama cli -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
Use pre-built binary 
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
# Run inference directly in the terminal:
./llama-cli -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
Build from source code 
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf gghfez/SmartMaid-123b-GGUF:Q2_K_M
Use Docker 
docker model run hf.co/gghfez/SmartMaid-123b-GGUF:Q2_K_M
Quick Links

Update: After further testing, this has turned out exactly like I wanted, and is one of my favorite models! It remains coherent at higher contexts and doesn't suffer the repetition issues I was having with Lumimaid.

GGUF Quants of gghfez/SmartMaid-123b

SmartMaid-123b

This experimental model is a hybrid creation combining aspects of Mistral-Large-Instruct-2407 and Lumimaid-v0.2-123B using LoRA (Low-Rank Adaptation) on the mlp.down_proj module.

Model Details

  • Base Model: Mistral-Large-Instruct-2407
  • Influence Model: Lumimaid-v0.2-123B
  • Method: LoRA extraction from Lumimaid and targetted application to Mistral-Large
  • LoRA Configuration:
    • Rank: 32
    • Alpha: 64
    • Target Module: mlp.down_proj

Process

  1. A LoRA was extracted from Lumimaid-v0.2-123B
  2. This LoRA was then applied to a fresh instance of Mistral-Large-Instruct-2407 targetting only the mlp.down_proj modules.
  3. The resulting model was merged to create this standalone version.

Purpose

The aim of this model is to incorporate the enhanced prose qualities of Lumimaid-v0.2-123B while retaining the core intelligence and capabilities of Mistral-Large. By applying the LoRA to the mlp.down_proj module, we sought to influence the model's language generation style without significantly altering its underlying knowledge and reasoning abilities.

Prompt Template 

<s>[INST] {input} [/INST] {output}</s>
Downloads last month
150
GGUF
Model size
123B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for gghfez/SmartMaid-123b-GGUF

Base model

NeverSleep/Lumimaid-v0.2-123B
Quantized
(12)
this model