poolside

/

Laguna-S-2.1-INT4

like 27

Follow

Poolside 1.54k

Hi there! Thanks for your comment.

We'll upload a new INT4 checkpoint in the next few hours that, in our internal tests, we've been unable to make loop.

I'll post here once it's uploaded.

Hi, thank you :) In the meantime I temporarily increased the repetition/presence penalties it seems to be good or maybe it is a coincidence, the repetition was less present on the int4, it is much easier to reproduce in nvfp4

Hi there! The updated INT4 weights are on the repo now. If you have any problems with them (or any feedback at all), please let us know: if I don't hear from you in the next 24 hours or so, I'll close this issue :)

Following up, does the DFlash Speculator also need to be updated? Acceptance rates have crashed to 0% after deploying the updated model.

Same here, I have 0 acceptance with the updated weights.

It keeps looping on new weights. Confirmed commit hash is 67dbeda .

Here is docker compose file (Dflash is not worked on this weights so I commented out )

  vllm-laguna:
    image: vllm/vllm-openai:vllm-x86_64-cu13-0.25.1-7a33ba9
    restart: unless-stopped
    shm_size: '48gb'
    ports:
      - "8000:8080"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: ${QWEN_GPU_COUNT:-1}
              capabilities:
                - gpu
    environment:
      HF_HOME: /root/.cache/huggingface
      HF_TOKEN: ${HF_TOKEN}
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface
      TRANSFORMERS_CACHE: /root/.cache/huggingface
      VLLM_CACHE_ROOT: /root/.cache/huggingface
      VLLM_LOGGING_LEVEL: ${VLLM_LOGGING_LEVEL:-info}
    volumes:
      - /mnt/data/models/vllm:/root/.cache
    security_opt:
      - seccomp=unconfined
    command:
      - --model
      - poolside/Laguna-S-2.1-INT4
      - --served-model-name
      - laguna-s2.1
      - --trust-remote-code
      - --max-model-len
      - "262144"
      - --enable-auto-tool-choice
      - --tool-call-parser
      - "poolside_v1"
      - --reasoning-parser
      - "poolside_v1"
      - --max-num-seqs
      - "16"
      - --gpu-memory-utilization
      - "0.95"
      - --cpu-offload-gb
      - "0"
      - --default-chat-template-kwargs
      - '{"enable_thinking":true}'
      - --override-generation-config
      - '{ "temperature": 0.7, "top_p":0.95}'
        # - --speculative-config
        # - '{"model":"poolside/Laguna-S-2.1-DFlash-INT4","num_speculative_tokens":15,"method":"dflash"}'

@TOSUKUi , just pointing out that the documentation specifies temp 0.7 top_p 0.95 only for the 1M context configuration. For your 256K configuration the config defaults to 1.0/1.0.

still looping after new weight, vllm nightly 0.23.1rc1.dev1432+g0231dd546

@jakehlee Thank you.

My understanding from README is that temperature <= 0.7 and top_p <= 0.95 are recommended when using the extended 1M-token context window, while temperature = 0.7 and top_p = 0.95 are recommended for general use.

I think this because the first guidance is in the Context Length section, while the second is in the Recommended Sampling section. Since both sections are at the same level under Usage, they seem to describe different use cases.

I also still have looping, I just removed the temperature settings but nothing changed.

        args:
            - --model
            - /models/Laguna-S-2.1-INT4
            - --served-model-name
            - laguna
            - --reasoning-parser
            - poolside_v1
            - --tensor-parallel-size
            - "4"
            - --max-num-batched-tokens
            - "2048"
            - --gpu-memory-utilization
            - "0.88"
            - --max-num-seqs
            - "8"
            - --max-model-len
            - "262144"
            - --enable-auto-tool-choice
            - --tool-call-parser
            - poolside_v1
            - --default-chat-template-kwargs
            - '{"enable_thinking": true}'
            - --enforce-eager

I think it’s useless to try, Ive retested the nvfp4 and it’s the same, all models are bad, when it doesn't go to infinity it can't even get out a simple HTML code he writes noob errors, for exemple for html comments he writes <!--..., in JS and PHP he uses the wrong concat symbols..., I don't know if it’s an issue in all quantisations or on the base model but for now it's impossible to do anything...

i posed in the Great Model chat but i havnt had any issues with mine??

here is the correct config also runs great on vllm/vllm-openai-rocm:latest but im also on gpu? have you tried a different vllm image? my 0.24.0 is 27 days old but it still works

services:
vllm-laguna:
image: aml731/vllm-aiter:v0.24.0
container_name: vllm-laguna
restart: unless-stopped
network_mode: host
ipc: host
group_add:
- "44"
- "993"
cap_add:
- SYS_PTRACE
security_opt:
- seccomp=unconfined
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
volumes:
- /home/models/gpu/poolside:/models
- /home/models/gpu/poolside/vllm-cache:/root/.cache
environment:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_ALLOW_RDNA4_AITER_ATTENTION: "1"
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION: "1"
VLLM_ROCM_USE_AITER_MHA: "1"
PYTORCH_ALLOC_CONF: "expandable_segments:True"
HIP_VISIBLE_DEVICES: "0,1,2,3"
command: >
python3 -m vllm.entrypoints.openai.api_server
--host 0.0.0.0 --port 8000
--model /models/Laguna-S-2.1-INT4
--served-model-name laguna-s-2.1
--trust-remote-code
--tensor-parallel-size 4
--dtype auto
--attention-backend ROCM_AITER_UNIFIED_ATTN
--compilation-config '{"pass_config":{"fuse_norm_quant":false}}'
--default-chat-template-kwargs '{"enable_thinking": true}'
--gpu-memory-utilization 0.95
--max-num-batched-tokens 8192
--max-model-len auto
--max-num-seqs 1
--enable-prefix-caching
--reasoning-parser poolside_v1
--tool-call-parser poolside_v1
--enable-auto-tool-choice
--safetensors-load-strategy=prefetch
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}'

It will work with the NVFP4 drafter

I tried everything: thinking off, attention_factor 1.0, capping thinking runs at 6K output tokens to reduce
the lost looping time, presence penalty, etc. The model loops in every configuration, just differently:

• Thinking off: degenerate action loops in agent use — e.g. the identical sed -n '2168,2185p' command issued
  255 times in a row across turns.
• Thinking on: occasional runaway reasoning inside a single response, growing until it fills whatever
  max_tokens allows (observed 61K characters of reasoning in one turn).
• Thinking on, long context: on a harder task it worked normally for ~130 turns, then collapsed into echo
  "test" repeated 181 consecutive times until the 262K context was full.

Setup: official INT4 checkpoint (revision 67dbeda), vLLM 0.25.1, TP4 on Ampere, your recommended sampling
(temperature 0.7, top_p 0.95, top_k 20), with and without the DFlash draft (identical behavior — it's the
target model). No way to get this working reliably; without external safeguards it can burn GPU at full
throttle for hours producing nothing. It's a pity, because the model is incredibly good when it works —
SWE-bench-style agent tasks solve cleanly in 30–80 steps when it doesn't degenerate — and I would love to
use it. Hoping for updated weights; happy to share full trajectories if useful.

So far the RC2 revision seems to be behaving better for me - I have both enable_thinking and preserve-thinking on.

@jakehlee , could you please share your configuration settings?

@BuiDoan here are my current vLLM flags, please note that I am also enabling the V2 model runner with VLLM_USE_V2_MODEL_RUNNER=1. This is running on 2xA100. Also, the acceptance rate for speculative config is near-zero because those weights have not been updated yet.

    --async-scheduling
    --model poolside/Laguna-S-2.1-INT4
    --revision RC2
    --tensor-parallel-size 2
    --max-model-len 128000
    --attention-backend AUTO
    --enable-auto-tool-choice
    --tool-call-parser poolside_v1
    --reasoning-parser poolside_v1
    --default-chat-template-kwargs '{
        "enable_thinking": true,
        "preserve_thinking": true
    }'
    --speculative-config '{
        "model": "poolside/Laguna-S-2.1-DFlash-INT4",
        "revision": "refs/pr/8",
        "num_speculative_tokens": 6,
        "method": "dflash"
    }'
    --override-generation-config '{
        "temperature": 0.7,
        "top_p": 0.95
    }'
    --trust-remote-code
    --enable-chunked-prefill
    --enable-prefix-caching
    --max-num-batched-tokens 8192

Mitigation that worked here: presence_penalty 2.0 removes the runaway-thinking tail

A presence penalty of 2.0 eliminated the long tail of runaway in-response reasoning on my host.
It is a sampling mitigation, not a fix, and it costs DFlash acceptance. Numbers below.

Setup: 4x RTX 3090 (24 GB), TP4, vLLM 0.25.1, target 67dbeda + draft f6b32f4
(num_speculative_tokens=4), mini-swe-agent 2.3.0 on SWE-bench Multilingual task
fmtlib__fmt-3750, temperature 0.7 / top_p 0.95 / top_k 20, thinking on with
preserve_thinking, 32K output cap and no separate thinking cap. Every row is N independent
rounds of the same task, so the spread is run-to-run variance only.

The sweep

Wall-clock spread collapses 8.1x -> 1.7x and solve rate goes up. Two further confirmation
rounds since: 7/7 passing, all under ~11 minutes. Decode rate steadies too, 27.7-131.4 tok/s
at baseline versus 98.6-110.6 at 2.0.

It does not decode faster -- it emits fewer tokens. Best round against best round: same 60-61
steps, 44,770 -> 34,001 completion tokens (-24%), 8.5 -> 6.5 min, and no 69-minute outlier
anywhere in the five rounds.

Cost: acceptance falls as the penalty rises

Relevant to the acceptance reports upthread:

presence_penalty   0     0.8    1.0    2.0
acceptance        53%   53.5%  41.9%  34.3%
accept. length   3.13   3.14   2.68   2.37    (of a possible 5.0 at 4 draft tokens)

Expected -- the penalty distorts the target distribution each draft is verified against, so more
drafts get rejected. If you lean on DFlash for throughput, that is the trade. Here the token
savings outweighed it easily.

Acceptance is a loop detector, not a target

The most useful thing I found. Looping inflates acceptance, because repetitive text is
trivially draftable. Within the same baseline run:

round-02   8.5 min, clean         53.2%
round-03  20.0 min                55.3%
round-04  69.3 min, the bad one   69.2%   <- highest acceptance, worst round

So that configuration's "63.7% average" was mostly the loop dominating the sample. Practical use:
a rising acceptance rate mid-run flags a loop in seconds instead of after 69 minutes.

journalctl -f | grep --line-buffered "Avg Draft acceptance rate"
curl -s http://127.0.0.1:PORT/metrics | grep -E "^vllm:spec_decode_num_(draft|accepted)_tokens_total"

Config

temperature: 0.7
top_p: 0.95
top_k: 20
presence_penalty: 2.0
enable_thinking: true
preserve_thinking: true

I had been capping responses to truncate runaway reasoning; that is no longer needed. Suppressing
the loop beats truncating it -- a truncated response costs a full generation and, with
preserve_thinking, feeds the looped text back as context on the retry.

Caveats

• One task, one host, N of 5-7 at the chosen value. A reproducible observation, not a benchmark.
• 2.0 is aggressive and departs from the checkpoint's authoritative generation_config.json.
  It penalises every already-emitted token equally, and code legitimately repeats identifiers and
  syntax. No harm to solve rate here, but I would not assume that generalises.
• It applies per response only -- vLLM builds the mask from the current completion's output
  tokens, not the prompt -- so it resets each API call. It suppresses within-response reasoning
  loops and does nothing for cross-step agent loops.
• Measured on 67dbeda, not a6977d0.

Separately: the 1M-native checkpoint needs more than 96 GB

I could not repeat this on a6977d0: 93 GB against 4x24 GB forces 40 GB of weight offload to
host RAM, and decode drops to **6 tok/s from ~100**, GPUs pegged at 100% stalled on PCIe. Worth
knowing if you are on 4x24 GB and wondering why the update crawls.

One encouraging datapoint from that attempt: draft 69fae8d against a6977d0 measured ~64%
acceptance versus 34.3% for f6b32f4+67dbeda at the same penalty. The updated pair looks
substantially better matched. Full sweep when I have the VRAM.

Raw per-round JSON available if useful.