---
base_model:
- bamec66557/MISCHIEVOUS-12B-Mix_0.1v
- bamec66557/MISCHIEVOUS-12B
library_name: transformers
tags:
- mergekit
- merge

---
# merge

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

## Merge Details
### Merge Method

This model was merged using the SLERP merge method.

### Models Merged

The following models were included in the merge:
* [bamec66557/MISCHIEVOUS-12B-Mix_0.1v](https://huggingface.co/bamec66557/MISCHIEVOUS-12B-Mix_0.1v)
* [bamec66557/MISCHIEVOUS-12B](https://huggingface.co/bamec66557/MISCHIEVOUS-12B)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
slices:
  - sources:
      - model: bamec66557/MISCHIEVOUS-12B-Mix_0.1v
        layer_range: [0, 40]
      - model: bamec66557/MISCHIEVOUS-12B
        layer_range: [0, 40]

    parameters:
      t:
        - filter: self_attn
          value: [0.1, 0.3, 0.7, 0.9, 1.0]  # 극적인 변화를 위한 급격한 증가
        - filter: mlp
          value: [1.0, 0.7, 0.4, 0.1, 0.0]  # 반대로 급격히 감소
        - filter: layer_norm
          value: [0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7,  # 첫 10개 레이어
                  0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3,  # 나머지 30개 레이어
                  0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3,
                  0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]
        - value: 0.9  # 기본 병합 비율을 높게 설정

merge_method: slerp  # 병합 방식을 slerp로 유지

base_model: bamec66557/MISCHIEVOUS-12B-Mix_0.1v  # 기본 모델

dtype: bfloat16  # 빠른 병합을 위한 데이터 타입

# 추가 옵션
regularization:
  - method: l2_norm  # L2 정규화로 병합 후 안정화
    scale: 0.005  # 정규화 강도를 낮춰 변화 허용

postprocessing:
  - operation: smoothing  # 병합 후 가중치 부드럽게 조정
    kernel_size: 5  # 커널 크기 증가로 더 넓은 범위 스무딩
  - operation: normalize  # 병합 후 정규화

```