JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT4

Qwen3 4B Thinking 2507 Heretic CodeFeedback — OpenVINO INT4

This repository contains the OpenVINO INT4 export of:

JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback

The source model is a merged code-focused fine-tune based on:

JoaoZaokk/Qwen3-4B-Thinking-2507-MiniMax-M2.1-Distill-heretic

This version was converted from the full merged safetensors model to OpenVINO IR with INT4 weight compression, intended primarily for Intel Arc / OpenVINO / OpenVINO GenAI inference.

This is my first merged model, thats basically for testing, i'll update it when i have time to.

Related repositories

Repository	Description
JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback	Master merged safetensors model
JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT4	OpenVINO INT4 export
JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT8	OpenVINO INT8 export
JoaoZaokk/Qwen3-4B-Thinking-2507-MiniMax-M2.1-Distill-heretic	Heretic source/base lineage model

Format

Item	Value
Source model	JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback
Format	OpenVINO IR
Weight compression	INT4
Group size	128
Ratio	1.0
Intended runtime	OpenVINO / OpenVINO GenAI
Tested device	Intel Arc A750 8 GB
Main use	Local code-focused inference on Intel GPU

Conversion command

optimum-cli export openvino \
  --model ~/models-src/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback \
  --task text-generation-with-past \
  --weight-format int4 \
  --group-size 128 \
  --ratio 1.0 \
  --trust-remote-code \
  ./Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT4

During conversion, most weights were compressed to INT4:

int4_asym, group size 128

A small portion may remain in INT8 depending on OpenVINO/NNCF layer handling.

Tested local inference

This export was tested with OpenArc / OpenVINO GenAI on:

Component	Value
GPU	Intel Arc A750
VRAM visible	~7.54 GiB
Runtime	OpenVINO GenAI
Engine	ovgenai
Device	GPU.0
Host environment	Ubuntu 24.04 VM with Intel Arc passthrough

Example observed metrics from a short Python-code prompt:

Metric	Value
Load time	16.89 s
TTFT	0.18 s
Prefill throughput	164.17 tokens/s
Decode throughput	28.66 tokens/s
TPOT	34.89 ms/token

OpenArc load example

curl -X POST http://localhost:8000/openarc/load \
  -H "Content-Type: application/json" \
  -d '{
    "model_path": "/models/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT4",
    "model_name": "Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT4",
    "model_type": "llm",
    "engine": "ovgenai",
    "device": "GPU.0",
    "runtime_config": {}
  }'

Training background

The source model was trained with QLoRA/LoRA on Python and code instruction datasets, then merged back into the base model.

Dataset	Samples used	Notes
iamtarun/python_code_instructions_18k_alpaca	5,000	Python instruction/code examples
m-a-p/CodeFeedback-Filtered-Instruction	5,000	Code instruction and feedback examples

A SWE-smith trajectory experiment was tested separately, but it was not used in the final merged version.

LoRA configuration of source model

Parameter	Value
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Sequence length	2048
Epochs per stage	1
Quantized loading during training	4-bit NF4
Trainable parameters	~33M
Trainable percentage	~0.81%

Target modules:

• q_proj
• k_proj
• v_proj
• o_proj
• gate_proj
• up_proj
• down_proj

Intended use

This INT4 OpenVINO version is intended for:

• local Intel Arc inference
• OpenVINO GenAI experiments
• Python code generation
• code explanation
• simple debugging
• instruction-following tests
• low-VRAM local inference compared to the F16 merged model

Hardware notes

Hardware	Expected status
Intel Arc A750 8 GB	Tested working
Intel Arc A770 16 GB	Expected better headroom
Intel Flex / Data Center GPU	May work if OpenVINO sees the GPU
CPU-only	Possible but slower
NVIDIA CUDA runtimes	Use the original safetensors, AWQ, GPTQ, or GGUF instead

Important notes

This is not the original F16 model. This is an OpenVINO INT4 compressed export.

For further conversions, use the original merged safetensors repository as the master source:

JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback

This is an experimental model. It may produce incorrect code, unsafe suggestions, or hallucinated explanations. Outputs should be reviewed before use in production or security-sensitive environments.