---
license: mit
base_model:
  - Qwen/Qwen2.5-Coder-1.5B
pipeline_tag: text-generation
library_name: transformers
tags:
  - pharo
  - smalltalk
  - code
  - code-completion
  - fim
  - sft
---

# Qwen2.5-Coder for Pharo Code Completion

This repository provides a Pharo-specialized variant of **Qwen2.5-Coder**, fine-tuned for fill-in-the-middle (FIM) code completion in **Pharo**, a Smalltalk-inspired, low-resource programming language.

The model is designed to complete partially written Pharo methods by predicting the missing code between a given prefix and suffix, making it suitable for IDE-assisted code completion.

## Model Details

- **Base model:** Qwen2.5-Coder
- **Model type:** Decoder-only causal language model
- **Programming language:** Pharo / Smalltalk
- **Primary task:** Fill-in-the-middle (FIM) code completion
- **Training pipeline:** Continued pre-training followed by supervised fine-tuning
- **Maximum context length:** 32,768 tokens

## Motivation

Large language models perform remarkably well on widely used programming languages such as Python, Java, and JavaScript, but their performance remains limited for low-resource languages such as Pharo.

Pharo inherits several unique characteristics from Smalltalk, including keyword-based message syntax, message-oriented control flow, and concise methods. These models are specifically adapted to these characteristics to provide more accurate and practical code completion while remaining compact enough for efficient local inference and IDE integration.

## Training Overview

The model is trained using a two-stage specialization pipeline.

### Continued Pre-Training

The base Qwen2.5-Coder model is first adapted to Pharo source code through continued pre-training on a curated corpus of Pharo methods. This stage teaches the model Pharo syntax, APIs, and coding conventions.

### Supervised Fine-Tuning

The adapted checkpoint is then fine-tuned on realistic fill-in-the-middle completion examples generated from Pharo methods. Completion targets begin at arbitrary cursor positions, better reflecting real-world code editing scenarios.

## Usage

Install the required libraries:

```bash
pip install -U "transformers>=4.37.0" accelerate torch
```

Load the model:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "pharo-llm/Qwen2.5-Coder-7B-SFT"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
).eval()
```

### Example

The model is trained for **fill-in-the-middle (FIM)** code completion using the Qwen FIM prompt format.

```python
prefix = """Collection >> selectEven

    ^ self select: [ :each |
"""

suffix = """
]
"""

prompt = (
    "<|fim_prefix|>"
    + prefix +
    "<|fim_suffix|>"
    + suffix +
    "<|fim_middle|>"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=16,
    do_sample=False,
)

completion = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(completion)
```

Expected completion:

```smalltalk
each even
```

## Paper

For additional details about the training methodology, datasets, and evaluation, see:

> **Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo**  
> https://huggingface.co/papers/2607.04939

## **Citation**

If you use this model in your research or applications, please cite our paper:

```bibtex
@article{teachingllmslowresourcelanguage,
  title={Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo},
  author={Kilian Kier and Alessandro Giagnorio and Omar AbedelKader and Oleksandr Zaitsev and Robert Peharz and Romain Robbes and Gabriele Bavota and Stéphane Ducasse},
  journal={arXiv},
  year={2026},
  doi={10.48550/arXiv.2607.04939}
}
```