Add metadata and paper reference
#2
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,3 +1,66 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
library_name: transformers
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
---
|
| 6 |
+
|
| 7 |
+
<img src="figs/logo.png" alt="EAGLE" width="220" align="left"><div align="center"><h1> EAGLE</h1></div>
|
| 8 |
+
|
| 9 |
+
<p align="center">
|
| 10 |
+
| <a href="https://arxiv.org/pdf/2401.15077.pdf"><b>Paper (EAGLE)</b></a> |
|
| 11 |
+
<a href="https://arxiv.org/pdf/2406.16858"><b>Paper (EAGLE-2)</b></a> |
|
| 12 |
+
<a href="https://sites.google.com/view/
|
| 13 |
+
eagle-llm"><b>Blog</b></a> |
|
| 14 |
+
</p>
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
<p align="center">
|
| 18 |
+
<a href="">
|
| 19 |
+
<img src="https://img.shields.io/badge/Version-v2.0.0-orange.svg" alt="Version">
|
| 20 |
+
</a>
|
| 21 |
+
<a href="https://opensource.org/licenses/Apache-2.0">
|
| 22 |
+
<img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg" alt="License">
|
| 23 |
+
</a>
|
| 24 |
+
<a href="https://github.com/SafeAILab/EAGLE/issues">
|
| 25 |
+
<img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" alt="Maintenance">
|
| 26 |
+
</a>
|
| 27 |
+
<a href="https://github.com/SafeAILab/EAGLE/pulls">
|
| 28 |
+
<img src="https://img.shields.io/badge/Contributions-welcome-brightgreen.svg?style=flat" alt="Contributions welcome">
|
| 29 |
+
</a>
|
| 30 |
+
</p>
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
##
|
| 34 |
+
|
| 35 |
+
<p align="center">
|
| 36 |
+
<img src="./figs/eagle2_t0.jpg" alt="benchmark" width="790">
|
| 37 |
+
</p>
|
| 38 |
+
|
| 39 |
+
EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a new baseline for fast decoding of Large Language Models (LLMs) with provable performance maintenance. This approach involves extrapolating the second-top-layer contextual feature vectors of LLMs, enabling a significant boost in generation efficiency.
|
| 40 |
+
|
| 41 |
+
This repository contains the model for the paper [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://huggingface.co/papers/2503.01840). The code can be found at https://github.com/SafeAILab/EAGLE.
|
| 42 |
+
|
| 43 |
+
- EAGLE is:
|
| 44 |
+
- certified by the <a href="https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md"><b>third-party</b></a> evaluation as the **fastest** speculative method so far.
|
| 45 |
+
- achieving **2x** speedup on <a href="https://github.com/pytorch-labs/gpt-fast"><b>gpt-fast</b></a>.
|
| 46 |
+
- **3x** faster than vanilla decoding (13B).
|
| 47 |
+
- **2x** faster than <a href="https://lmsys.org/blog/2023-11-21-lookahead-decoding/"><b>Lookahead</b></a> (13B).
|
| 48 |
+
- **1.6x** faster than <a href="https://sites.google.com/view/medusa-llm"><b>Medusa</b></a> (13B).
|
| 49 |
+
- provably maintaining the consistency with vanilla decoding in the distribution of generated texts.
|
| 50 |
+
- trainable (within 1-2 days) and testable on 8x RTX 3090 GPUs. So even the GPU poor can afford it.
|
| 51 |
+
- combinable with other parallelled techniques such as vLLM, DeepSpeed, Mamba, FlashAttention, quantization, and hardware optimization.
|
| 52 |
+
|
| 53 |
+
EAGLE-2 uses the confidence scores from the draft model to approximate acceptance rates, dynamically adjusting the draft tree structure, which further enhances performance.
|
| 54 |
+
|
| 55 |
+
- EAGLE-2 is:
|
| 56 |
+
- **4x** faster than vanilla decoding (13B).
|
| 57 |
+
- **1.4x** faster than EAGLE-1 (13B).
|
| 58 |
+
|
| 59 |
+
<p align="center">
|
| 60 |
+
<img src="./figs/eagle2.gif" alt="demogif">
|
| 61 |
+
</p>
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
Using EAGLE-2, the inference speed on 2 RTX 3060 GPUs can be faster than vanilla autoregressive decoding on an A100 GPU.
|
| 65 |
+
|
| 66 |
+
[Remaining content as is...]
|