---
license: apache-2.0
language:
- en
- fr
- ro
- de
- multilingual
base_model:
- google/flan-t5-small
pipeline_tag: text-generation
tags:
- loom
- text2text-generation
library_name: loom-py-rt
---
# FLAN-T5 Small

Google's instruction-tuned T5-small, exported for loom.cpp. Family 6: text in, text out, through an encoder read once and a KV-cached decoder that cross-attends to it -- the first encoder-decoder TEXT model and the first SentencePiece Unigram vocabulary in this collection.

This is a [loom.cpp](https://github.com/loom-ai-org/loom.cpp) export: a single self-describing GGUF
that carries its own graph topologies, tokenizer (if any) and driver script, produced by
[loom-exporter](https://github.com/loom-ai-org/loom-exporter).

## Original model

Exported from [`google/flan-t5-small`](https://huggingface.co/google/flan-t5-small). Weights are unmodified; this repo packages the same parameters into
loom.cpp's GGUF format.

## License

`apache-2.0`, inherited from the base model above.

## Language(s)

`en`, `fr`, `ro`, `de`, `multilingual`

## Usage

Run it with [loom-py](https://github.com/loom-ai-org/loom-py) -- `loom-py-rt` on PyPI:

```sh
pip install -U "loom-py-rt[hub]"
```

```python
import loom

model = loom.Model.from_pretrained("loom-ai-org/flan-t5-small-loom")

# The instruction is part of the text -- this is an instruction-TUNED encoder-decoder, not a chat
# model, and it has no template to apply.
print(model.text2text.infer(
    "translate English to German: The weather today is cold and rainy in the north.",
    max_new_tokens=32))

# The same door, on the other tasks this checkpoint was tuned for:
print(model.text2text.infer("Answer the following question. What is the capital of France?"))
print(model.text2text.infer(
    "summarize: The committee met on Tuesday to discuss the annual budget. After three hours of "
    "debate, the members agreed to postpone the vote until the following month, citing incomplete "
    "figures from the finance office.",
    max_new_tokens=64))
```

### The layer underneath

The call above is the high-level door: one per task, named for the modality pair it maps between, with
the windowing, sampling and assembly this model needs already applied. Under it, `model.infer(...)`
passes your arguments straight to the driver this GGUF embeds -- which is where you go for a knob the
door does not name.

`model.driver_source` prints that driver, including a header comment documenting every argument it
accepts for this model, and is the authority on it. See [loom-py](https://github.com/loom-ai-org/loom-py) for the API and
[loom.cpp](https://github.com/loom-ai-org/loom.cpp) for what the engine does between the two.

## Known limitations

**It is 80M parameters, and it answers like it.** flan-t5-small is the smallest member of the flan collection: it follows an instruction's SHAPE reliably and is often wrong about the content. Asked for the capital of France it says `london`, and `transformers` says the same thing on the same checkpoint -- that is the model, not the export. Use it to try the interface, and step up to `flan-t5-base` or larger for answers you intend to keep.

**The instruction is part of the text.** There is no chat template and no system turn; prompts look like `translate English to German: ...` or `Answer the following question. ...`, which is how the checkpoint was tuned. A bare sentence with no instruction is out of distribution and usually comes back as a fragment of itself.

**Greedy, not beam search.** The upstream `config.json` names 4 beams for its translation and summarization presets. The export decodes one token at a time from the argmax (or from the sampler, if you pass `temperature=`), so translations here are the greedy path through the same model rather than what the published presets would give.

Sequences are capped at 512 tokens on each side. T5 has no learned position table, so nothing in the file stops a longer source -- but its relative-position buckets saturate at 128 tokens of distance and the KV cache is built at 512, which is the length this export is honest about.

## Files

- `flan-t5-small.gguf` -- the model, exported with loom-exporter.