LRieser/steam-ai-mention-qwen3-8b-lora

Steam AI-mention classifier (Qwen3-8B LoRA adapter)

LoRA adapter for Qwen/Qwen3-8B that turns the model into a classifier of Steam game reviews: given the review and its game context, it answers whether the review discusses generative AI used to produce the game's content (art, voice, music, writing, code, translation). The answer is a JSON object. Trained on 19,152 Steam reviews labeled by a large language model. Multilingual.

The same weights merged into the base model are available as LRieser/steam-ai-mention-qwen3-8b-merged (no base download, loads directly into vLLM).

Output

{"mentions_genai": true, "confidence": "high", "reasoning": "Calls the portraits 'AI slop'."}

Field	Values
mentions_genai	true if the review discusses generative AI in the game's production, else false
confidence	"low", "medium", "high"
reasoning	one or two sentences quoting the decisive phrase

true covers explicit terms ("AI art", "AI-generated", "AI slop"), tool and vendor names, their equivalents in other languages, reactions to a developer's AI disclosure, and speculative claims when the context supports a genuine suspicion. false covers gameplay AI (enemy or NPC behaviour), procedural generation, the reviewer's own use of AI tools, and rhetorical comparisons ("an AI could have written this"). The full rubric is the system prompt in prompt_system.md.

Usage

The model expects the system prompt from prompt_system.md and a user message in the format of prompt_user_template.md (game name, genres, tags, release date, the developer's AI disclosure if any, review date, language, recommendation, and the review text). Both files are in this repository.

import json
import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "LRieser/steam-ai-mention-qwen3-8b-lora"
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, repo).eval()

system_prompt = open(hf_hub_download(repo, "prompt_system.md"), encoding="utf-8").read()

user_message = """=== GAME ===
name: Example Quest
genres: RPG, Indie
top_tags: Pixel Graphics, Story Rich, 2D
release_date: 2024-09-12
early_access: False

=== DISCLOSURE ===
  flag: no

=== REVIEW ===
written_on: 2025-03-02
regime: post_mandate_2024  (computed from effective date = max(written, edited))
language: english
voted_up: False
text: |
The portraits are AI slop, every face has that same plastic look."""

messages = [{"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=128, do_sample=False)
result = json.loads(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
result["mentions_genai"]  # True

Plain generate() runs at about one review per second on an RTX 4090. For corpus-scale inference use the merged weights with vLLM (about 11 reviews per second).

Training

• Base model: Qwen/Qwen3-8B.
• Method: QLoRA with DoRA (use_dora=True), rank 16, alpha 32, dropout 0, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. NF4-quantised base, bf16 compute.
• Data: 19,152 Steam reviews, 4,585 positive. Two sampling strata: reviews matching a multilingual generative-AI keyword list (14,379) and a uniform random sample of reviews (4,773). Labels were produced by DeepSeek V4-Pro in thinking mode under the rubric in prompt_system.md, after a 250-review audit that refined the rubric. Twenty languages; the largest are English (55%), Simplified Chinese (12%), Russian (10%), Italian (6%), and German (4%).
• Optimisation: paged AdamW 8-bit, learning rate 2e-4, cosine schedule, warmup ratio 0.05, weight decay 0.01, gradient clipping 1.0, batch size 1 with gradient accumulation 4, gradient checkpointing, maximum sequence length 4096. Loss on the assistant JSON only.
• Schedule: 2 epochs (7,662 steps); the released adapter is the checkpoint with the lowest validation loss (0.295). Final token accuracy 93.9%.

Evaluation

Held-out test split of 960 reviews (seed 42), never used in training, scored with the merged bf16 weights under vLLM. The keyword-matched stratum (n=720) contains almost all positives; the random stratum (n=240) checks for false positives on ordinary reviews.

Metric	Value
Macro-F1, keyword-matched stratum	0.953
Positive-class precision / recall / F1	0.959 / 0.912 / 0.935
Macro-F1, English / non-English	0.968 / 0.940
False positives, random stratum (n=240)	0
Throughput (RTX 4090, vLLM bf16, merged)	about 11 reviews/s

The companion encoder (LRieser/steam-ai-mention-mmbert-base) reaches macro-F1 0.925 on the same split at about 430 reviews per second.

Citation

@misc{rieser2026steamaimentionqwen3lora,
  author    = {Rieser, Lars and Ohlrogge, Fynn and Joshi, Anant and Sethi, Navneet},
  title     = {Steam AI-mention classifier (Qwen3-8B LoRA adapter)},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/LRieser/steam-ai-mention-qwen3-8b-lora}
}