HassanB4/aragenre-combined-generalist-equalweight

AraGenre Combined Generalist — Equal-Weight Ensemble

An equal-weight majority-vote ensemble combining the project's best trained-pipeline model with all four models, for hierarchical Arabic genre classification. This is not a single fine-tuned checkpoint — the repo contains a weights JSON and a combination script, not trained weights of its own.

Authors: Hassan Barmandah (NAMAA Community; Umm Al-Qura University), Israa Elhosiny (NAMAA Community), Yousra El-Ghawi (NAMAA Community), Omer Nacar (NAMAA Community)

⚠️ Generalization Note

This is the best result anywhere in this project among the no-dev-tuning models — a genuine achievement, since it reaches 0.9504 hierarchical F1 on dev using equal weights, a fixed tiebreak rule, and no dev-label weight tuning at all.

That said, even this dev number is still a dev number, and it does not reflect real hidden-test-set performance. Per the project's system-description paper, this whole fine-tuned-encoder lineage — including this ensemble's trained-pipeline member — collapsed to 0.22–0.44 hierarchical F1 on the actual 27,972-item hidden test set (74 specific genres under 6 broad genres), because it was calibrated to a hand-augmented definition style and a much smaller taxonomy than the official test set uses.

The system that actually won for this team — 0.7013 hierarchical F1, 3rd of 18 teams on the official CodaBench leaderboard — was a separate, zero-shot DeepSeek-LLM pipeline with no fine-tuning at all (stage2_llm_zeroshot_pipeline/ in the project repo). This artifact is not that system.

Approach

Equal-weight majority vote over e5-large-mnrl-xgenre-phase1 (the best single model from the trained pipeline) plus all four models (mdeberta-nli-zeroshot, llm-zeroshot-reasoning, e5-large-paraphrase-augmented, e5-large-multicorpus-xgenre). Same no-dev-tuning philosophy as aragenre-generalist-ensemble-zeroshot-vote — a fixed tiebreak rule, no weight search — with one strong trained-pipeline member added to the vote. This is the "combine the generalized pipeline with the trained pipeline" counterpart to the 13-model ensemble that keeps the no-dev-tuning property.

Components

e5-large-mnrl-xgenre-phase1, mdeberta-nli-zeroshot, llm-zeroshot-reasoning, e5-large-paraphrase-augmented, e5-large-multicorpus-xgenre.

Training data

None directly — this is a weight/vote recipe over pre-scored component models. No dev labels were used in selecting weights.

Usage

Requires all 5 component scripts run first, then:

python combined_generalist_equalweight.py

See the project repository for the full script and component-model requirements.

Citation

If you use this work, please cite our system-description paper:

@inproceedings{barmandah-etal-2026-namaa,
  title = {NAMAA at AraGenre 2026: From Encoder Baselines to Self-Consistent LLM Ensembling for Hierarchical Arabic Genre Classification},
  author = {Barmandah, Hassan and Elhosiny, Israa and El-Ghawi, Yousra and Nacar, Omer},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

Please also cite the AraGenre 2026 shared task overview paper:

@inproceedings{elhaj-etal-2026-aragenre,
  title = {AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task},
  author = {El-Haj, Mo and Ezzini, Saad and Abudalfa, Shadi and Lamsiyah, Salima and Jarrar, Mustafa},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

License

Apache 2.0