Problemsby GlyphRunning vLLM locally hits memory and speed limits
21 posts from 20 people15d activeRising, +7 this week
Posts per day
The posts behind it
[Bug]: `enable_weights_track` refuses correct weight-only NVFP4 checkpoints (input scales that no export writes and no kernel reads)
### Your current environment The output of python collect_env.py (trimmed) `compressed-tensors==0.17.0`. The Python sources were those of main `f358f0578` (2026-11-06) on top of that nightly wheel; the files involved are the same in both and on main `3012587f0`. ### 🐛 Describe...
[Bug]: compressed-tensors loads checkpoint FP8 k/v scales into layers kept in BF16 by kv_cache_dtype_skip_layers, and FlashInfer applies them
### Your current environment The output of python collect_env.py ### 🐛 Describe the bug `unsloth/gemma-4-26B-A4B-it-NVFP4` declares a `kv_cache_scheme` with static FP8 k/v scales for all 30 attention layers. We serve it with: The 25 sliding-window layers are skipped, so they...
[Bug]: MiniMax-M3 sparse attention layers drop checkpoint FP8 KV-cache scales and run with scale 1.0
### Your current environment The output of python collect_env.py (trimmed) The Python sources were those of main `f358f0578` (2026-11-06) on top of that nightly wheel; the files involved are the same in both and on main `3012587f0`. ### 🐛 Describe the bug **Summary**...
[Feature]: Enable and validate pipeline parallelism for Domyn Edge
### 🚀 The feature, motivation and pitch ### Proposal I'd like to enable and validate pipeline parallelism (PP) for `domyn/Domyn-Edge-Base` and `domyn/Domyn-Edge-SFT`, following the native model support added in #62544. I'm opening this to confirm the scope and coordinate before...
[RFC]: A KV compatibility identity, enforced on NIXL and P2P transfers and readable by routers
### Motivation. vLLM transfers KV between engines over NIXL (prefill to decode) and the P2P offload tier (engine to engine). Each checks compatibility once per peer connection. NIXL hashes the vLLM version, model path, dtype, KV shape, attention backend, cache dtype and...
[RFC]: [Draft] Split FusedMoEQuantConfig into static schema and runtime state
# Fused MoE Static and Runtime Quantization Configs **Status:** Proposal / RFC **Date:** 2026-11-03 ## Summary Split `FusedMoEQuantConfig` into two objects with different lifetimes: * `FusedMoEQuantSchema` is an immutable, tensor-free description of the quantization scheme owned...
Quantizing gpt-oss on load with `Mxfp4Config()` crashes its forward and saves noise
### System Info - `transformers` main at 26754696fdbcdc356f07bf9762fc364923a9b71f, and 5.20.2 - `torch` 2.14, H100, `kernels` 0.17.2 ### Who can help? @SunMarc @MekkCyber ### Information - [ ] The official example scripts - [x] My own modified scripts ### Tasks - [ ] An...
[CI Failure]: MI300 MoE Kernels Multi-GPU - MoriEPv2 combine block_num 304 exceeds xdb_flag_slots 256 in test_modular_kernel_combinations.py
### Name of failing test `kernels/moe/test_modular_kernel_combinations.py::test_modular_kernel_combinations_multigpu[...-MoriEPv2PrepareAndFinalize-...-2]` Failing cases (Buildkite step `:amd: (MI300) MoE Kernels Multi-GPU`): -...
GPTQ dynamic ignores silently fail for Qwen3.5-family checkpoints: three module-name spaces
...dynamic entry was authored against a name form that never matches): Over-broad pattern (workaround attempt) proving the matcher itself works only in the module-tree space: (i.e. when a pattern DOES match in module-tree space, exclusion works - here too broadly; narrowing it to...
[RFC]: Bring vLLM's development cost back to linear: a small core with owned, independently tested components
### Motivation. Every new model, kernel, platform or quantization method in vLLM lands in the same tree, the same review queue and the same CI. Because components import each other directly, the cost of a change grows with the number of pairs that can interact, not with the...
[Bug]: `hadacore_transform` gives wrong results for odd row counts (INT4 per-token-head KV cache): garbage K/V and intermittent illegal memory access
...stayed intact. So the kernel appears to **read** past the end of the tensor. The set of failing N varies with what memory follows the bf16 copy made in `fast_hadamard_transform`. A likely place to look: for head_dim 256, `run_fht`...
[Question]: Qwen4Exp PLE n-gram table is materialised dense in bf16 — can it stay compressed in pinned host memory?
### Environment | | | |---|---| | vLLM | `0.32.0` | | Python | `3.12.15` | | torch | `2.13.0+cu130` | | transformers | `5.19.0` | | triton | `3.7.1` | | glq | `0.8.28` | | CUDA / driver | `13.0` / `595.91.07` | | GPU | 2x consumer Blackwell (`sm_120`), 31.85 GiB total | > **Note...
[Roadmap] vLLM Roadmap Q4 2026
In [#32455](https://github.com/vllm-project/vllm/issues/32455), we broke down vLLM's goals into various special interest groups (SIGs). Building on the [Q3 roadmap](https://github.com/vllm-project/vllm/issues/48168), please find below each SIG's Q4 roadmap. You can find regular...
$2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill
...to see if I could build a big VRAM rig with usable speed for relative peanuts.. I was struggling with llama.cpp for a long time, but the prefill was pretty bad (around 350-450 t/s average with this same model) and vLLM just didn't work on the cards. Plus llama.cpp just sucks at...
Feature Request: Decision 2.0 GGUF and systemone support
### Prerequisites Implementation: #30158 at `3c64a58`. Upstream checked: `c35b667`. I checked the README, contribution guidelines, existing issues, PRs and discussions and found no existing Decision 2.0 proposal. ### Feature Description Add native GGUF conversion and...
[ROCm][AMD] Qwen3.8-Flash-Next gfx950 / MI355X Performance Optimization
### Proposal to improve performance This issue is the gfx950 / MI355X performance optimization plan and tracker for `amd/Qwen3.8-Flash-Next-Quark-MXFP4` (`qwen4_exp`: GatedDeltaNet in 36 layers, QuerySparse Attention in 12, HyperConnection gated residual, MXFP4 MoE). The work...
[RFC]: A Reusable KV Compression Layer for Transfer and Storage
### Motivation. **Reference baseline:** vLLM v0.30.0 We propose an optional, reusable KV compression capability in vLLM for KV transfer and storage. KVServe manages the compression workflow, including algorithm composition, profile selection, and encoding/decoding. Mooncake P/D...
[RFC]: Online quantization from arbitrary precision
### Motivation. This RFC is ported from wider https://github.com/vllm-project/vllm/issues/52167. We propose to extend online quantization API to allow re-quantization from arbitrary weight original precision (arbitrary weight quant key), not only `BF16/FP16/FP32`. `FP8 -> MXFP4`...
MtpModel/MtpLayer topology is too rigid to represent non-uniform MTP architectures
### Feature request `MtpModel` (added in #46229) models MTP draft layers with a single fixed `MtpLayer` shape (`enorm` + `hnorm` + `eh_proj(2h->h)` + `mtp_block = decoder_layer_cls`, plus optional `post_norm`), and projects through `main_model.lm_head`. This fits...
[Feature]: Fast Start For vLLM
## Overview Fast Start reduces vLLM startup time by avoiding repeated checkpoint loading, weight processing, and kernel autotuning during engine restarts and scaling events. The core design uses a persistent per-GPU weight daemon. The daemon loads and processes model weights...
[Feature]: Consider B12X in automatic NVFP4 MoE backend selection on SM120/SM121 before the Marlin fallback
### 🚀 The feature, motivation and pitch vLLM main already contains the optional direct **B12X** NVFP4 linear *and* MoE backend merged in #52018. Its support predicate explicitly covers **SM120 and SM121**, NVFP4 W4A16, supported shapes, tensor-parallel constraints and package...
Companies and products named
About this problem
Evidence
21 posts from 20 people in 4 places, about 2 a week over 67 days. Mostly on vllm-project/vllm, huggingface/transformers, r/LocalLLaMA. Tools named alongside: GitHub, NVIDIA, Hugging Face, Oracle.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit
Summary, model-written
vLLM users encounter slow startup times, model weight loading bottlenecks, and quantization performance limitations across hardware architectures.
How it was grouped
Posts that state a pain and match the "local-llm" rule about vLLM. First post Sep 1, 2026, latest Nov 6, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running Ollama locally hits memory and speed limits
47 posts from 45 people on GitHub issues, Reddit.
Running llama.cpp locally hits memory and speed limits
45 posts from 41 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
22 posts from 21 people on GitHub issues, Reddit, Hacker News.
Running Qwen locally hits memory and speed limits
18 posts from 18 people on GitHub issues, Reddit, Stack Overflow.
Running Hugging Face locally hits memory and speed limits
17 posts from 15 people on GitHub issues, Reddit.
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
Running OpenAI locally hits memory and speed limits
13 posts from 13 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
8 posts from 8 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running PyTorch locally hits memory and speed limits
5 posts from 5 people on GitHub issues, Hacker News.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other vLLM problems
Trouble with vLLM draft
Top post: "Draft model construction resets data_parallel_index to 0 (load_draft_model / create_draft_parallel_config),", one of 3 posts from 3 people on GitHub issues.
vLLM users keep filing bugs
Top post: "Qwen3.8-flash-next 0% MTP acceptance rate in disaggregated PD serving", one of 34 posts from 31 people on GitHub issues, Reddit.
vLLM upgrades break working setups
17 posts from 17 people on GitHub issues.
vLLM vulnerabilities are hard to triage and patch
9 posts from 9 people on Glyph Ibis.
vLLM runs out of context on real work
5 posts from 5 people on GitHub issues.
vLLM GPU access and drivers are painful
5 posts from 3 people on GitHub issues.
Fine-tuning vLLM is costly and unpredictable
4 posts from 4 people on GitHub issues.
Trouble with vLLM cache
Top post: "AOT compile cache never loads for Qwen3.5 models: Kernel index 1 not found in id_to_kernel", one of 4 posts from 4 people on GitHub issues.
Trouble with vLLM kernel
Top post: "ROCM_AITER_MLA can select an aiter MLA decode kernel that silently returns uncorrelated output (gqa_ratio=16,", one of 3 posts from 3 people on GitHub issues.
Trouble with vLLM size
Top post: "EPLB counts load for the wrong expert when step_interval < window_size", one of 3 posts from 3 people on GitHub issues.
vLLM runs out of memory
3 posts from 3 people on GitHub issues.
vLLM fails to return valid structured output
3 posts from 3 people on GitHub issues.
History