Problemsby GlyphRunning Qwen locally hits memory and speed limits
18 posts from 18 people15d activeSteady, +2 this week
Posts per day
The posts behind it
[Bug]: Qwen3.5 vision RoPE angle precision differs from HF (public 9B reproduction, mixed FP32 workaround results)
### Your current environment Manually collected, relevant environment fields (not a full collect_env.py dump): The successful public run used the documented optional process-only `--disable-external-flash-attn` flag because the reused environment inherited a pip FlashAttention...
Linux video input for llama-server models
## Problem Video-capable models in the library (Qwen3-VL / Qwen3.5 GGUF, LFM3-VL, LFM3-Omni) accept MP4/MOV on macOS through the MLX runner (#19348). On Linux and Windows, and for every GGUF model, an MP4 passed in `images` reaches llama-server and fails with `Failed to load...
QuantizedCache.crop() raises IndexError across FP8, Quanto and HQQ, breaking speculative decoding
...| 8 | All three quantized backends successfully initialized and updated before failing during rollback. ### 2. Real pretrained Qwen2.5 reproduction **Target:** `Qwen/Qwen2.5-1.5B-Instruct` (`cuda:0`) **Assistant:** `Qwen/Qwen2.5-0.5B-Instruct` (`cuda:1`) **Observed across three...
Pulls from hf.co fail behind DNS64: the NAT64 addresses of the CDN are refused as non-public
### What is the issue? On a network with a DNS64 resolver, `ollama pull hf.co/...` never completes. The Hub redirects the download to a CDN, and Ollama refuses the CDN because of a NAT64 address: `ollama pull` keeps printing "pulling manifest" while the server retries each part....
[Bug]: [ROCm] AutoAWQ checkpoints never use the RDNA W4A16 kernel; AWQ decodes 2.5x slower than BF16 on gfx1151
### Your current environment - Image: official `docker.io/rocm/vllm:rocm10.1.0_ubuntu24.04_py3.14-pytorch_2.13.0_vllm-0.29.0` - vLLM: `v0.29.1.dev0+g98dff2a81` - Stack: PyTorch 2.13, ROCm 10.1 - Hardware: AMD Ryzen AI Max+ 395, Radeon 8060S (**gfx1151**, RDNA 3.5), 128 GB...
Why does my local Qwen LLM report that no business indicators are available when my pandas DataFrames contain data?
.... Kaggle with a Tesla T4 GPU. . The forecasting pipeline executes, but I'm having trouble with the final LLM analysis.. The problem My notebook contains several populated pandas DataFrames, including:. op : daily operational data. . fin : financial transaction data. . rv :...
Practical advice for browser/tool use with 4070 Ti SUPER (16GB VRAM) setup?
...experimenting with both Gemma 4 12B and Qwen 3.5 9B. . I'm looking for some practical advice on the best way to handle browser-based tool use and automation within this specific hardware environment. For those who have integrated local models with web-scraping or automated...
Misc. bug: Vulkan GPU device ordering affects PP performance
...processing speed is roughly half of what I was getting before I installed the 2nd GPU. **Workaround:** Forcing a flip in the enumeration of the Vulkan devices with the `GGML_VK_VISIBLE_DEVICES` environment variable restores prompt processing speed to original levels: Also...
[macOS/Metal] MLX SDPA kernel panic on M2 Max: Maximum threads per threadgroup is 896 but requested 960 (Qwen 3.8 / Gemma 4)
### What is the issue? Description: When running models with large attention head dimensions ($head\_dim = 256$, such as qwen3.8:27b or gemma4:31b) on Apple Silicon GPUs (M2 Max, 64GB RAM), Ollama’s compiled MLX SDPA backend panics during prefill execution due to a hardware...
Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix
...contract.. 512k is nearly free (≤8% on either engine). On a 32 GB 5090 experts are the bottleneck, not attention KV.. Prefix cache makes repeat long prompts free (~0.09 s warm re-prefill of a 39k prompt).. Quality: statistically tied - teacher-forced PPL 4.772 (Infernix) vs...
Qwen 3.8 Flash Next-GSQ-RCO-IQ2_XS at ~21 tok/s on just an RTX 3060 12GB + 16GB DDR4 RAM(No gate pruning, 100% bit-exact)
...running a 68GB MoE on a 16GB RAM machine with stock llama.cpp , you probably know how painful it gets.. I'm talking 1.4-2.1 tok/s , with over 1,500 major page faults per token in some runs. Linux ends up constantly pulling model data from the SSD because there's simply not...
What are you using for NVFP4 and Do you like it?
...using to run nvfp4 on multi-gpu?. the only thing i can get to run is unsloth and vllm is too slow and takes FOREVER to fucking load. TensorRT-LLM has too many issues, so what are people using?. And do you like nvfp4 vs q4 qguf for qwen 3.8? apples to oranges is nvfp4 closer to...
[Bug]: moe_backend="batched_triton" crashes engine startup on unquantized MoE model — TypeError: BatchedTritonExperts.__init__() missing 2 required positional arguments
### Your current environment ### 🐛 Describe the bug #### Description Selecting `moe_backend="batched_triton"` for an **unquantized** MoE model is accepted at config-validation time, but the engine crashes during model load with: The root cause is visible in...
I benchmarked Qwen 3.8 27B on my M4 Max - here's what I learned about local models, context, and coding performance
...for this All of this started with me manually sending Ollama API requests.. That got annoying quickly, so somewhere between staring at terminal output and wondering why my MacBook had turned into a very expensive space heater, I ended up turning the experiments into a CLI:....
Qwen 3.8-27 tips for my setup
...LM Studio but have since switched to VS Code with Cline. Could you please offer some advice on optimal settings for this model and, if possible, tips on how to best configure VS Code with Cline? I’d like to switch to Ollama and move away from LM Studio (hoping for a smoother...
UkisAI Swift Series / 27B, Flash Next and Bonsai 2 + GSQ-RCO / -63.4% thinking, x1.95 speed with xhigh accuracy
...a bug, but a simple matter of the Swift models not falling into overthinking loops and failing the task, rather pursuing it until the end, leading to higher average token usage. The token reduction still falls in the -38.7% range when compared apples-to-apples.. We also added a...
Qwen-3.8-27B is good enough that I stopped using API
...Qwen-3.8 and I had a rocky start, because it thinks so much. Watching it working is painful, so you have to stop doing that. You have to let it work unsupervised. And that's okay, because it really is able to complete complex refactors on its own, making good decisions along the...
Misc. bug: When KV is in RAM the MTP draft KV cache is also in RAM
### Name and Version version: 0.3.0-dev (build 10675, commit 90c26fcd4) built with MSVC 19.44.35228.0 for Windows AMD64 I've been able to reproduce on a7cc83bba ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-server ### Command line...
Companies and products named
About this problem
Evidence
18 posts from 18 people in 7 places, about 2 a week over 68 days. Mostly on r/LocalLLaMA, vllm-project/vllm, ollama/ollama. Tools named alongside: Ollama, GitHub, NVIDIA, llama.cpp.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit, Stack Overflow
How it was grouped
Posts that state a pain and match the "local-llm" rule about Qwen. First post Aug 31, 2026, latest Nov 6, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running Ollama locally hits memory and speed limits
47 posts from 45 people on GitHub issues, Reddit.
Running llama.cpp locally hits memory and speed limits
45 posts from 41 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
22 posts from 21 people on GitHub issues, Reddit, Hacker News.
Running vLLM locally hits memory and speed limits
21 posts from 20 people on GitHub issues, Reddit.
Running Hugging Face locally hits memory and speed limits
17 posts from 15 people on GitHub issues, Reddit.
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
Running OpenAI locally hits memory and speed limits
13 posts from 13 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
8 posts from 8 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running PyTorch locally hits memory and speed limits
5 posts from 5 people on GitHub issues, Hacker News.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other Qwen problems
Qwen users keep filing bugs
Top post: "Get ready for qwen4", one of 19 posts from 19 people on GitHub issues, Reddit, Community forums.
Qwen upgrades break working setups
5 posts from 5 people on GitHub issues.
Fine-tuning Qwen is costly and unpredictable
4 posts from 4 people on GitHub issues, Hacker News, Reddit.
Trouble with Qwen local
Top post: "Feature request: Local image-generation model support with hardware-tuned inference", one of 3 posts from 3 people on Reddit, Stack Overflow, GitHub issues.
Qwen rate limits and usage caps block work
2 posts from 2 people on Reddit, GitHub issues. Unverified.
Qwen runs out of context on real work
2 posts from 2 people on Reddit, GitHub issues. Unverified.
Qwen fails to return valid structured output
2 posts from 2 people on GitHub issues. Unverified.
History