Running NVIDIA locally hits memory and speed limits
17 posts from 17 people13d activeFading, +0 this week
Posts per day
The posts behind it
CUDA backend: fatal CUDA error + NVIDIA driver reset (nvlddmkm Event 153) during sustained batched prompt processing while an NVENC encoder shares the GPU — Vulkan build is stable at the same VRAM footprint (depth-dependent)
**Environment:** - NVIDIA GeForce RTX 4090 (24 GB), single GPU, driver 621.14 - Windows 26H2, build 26300 - tested at three servicing levels (Nov 2026 cumulative, Oct 2026 cumulative KB5127648, and the pre-October Sept 22 state); the crash reproduces at all three, only the...
Feature request: make mixed-vendor (e.g. Intel Vulkan target + NVIDIA CUDA draft) speculative decoding actually worth using
### Context Setup: Intel Arc Pro B65 (32 GB, primary) + NVIDIA RTX 3050 (6 GB, secondary) on the same host. Goal: use the smaller NVIDIA card purely as a speculative-decoding draft device for a target model running on the Intel card via Vulkan. This works today, but only after...
Compile bug: CUDA11 sm_50 fails in lightning-indexer.cu
### Git commit e30ad80fd7fa09a858bf8b8f4118e6479809486e ### Operating systems Windows ### GGML backends CUDA ### Problem description & steps to reproduce The problem was produced on a clean Linux x86_64 CUDA 11.7 build which was targeting `sm_50`. It failed in...
[Tracking] Replace Marlin with Humming across NVIDIA architectures
### Motivation. Humming now covers the quantization formats Marlin serves (INT4/INT8 GPTQ/AWQ, FP8, MXFP8, MXFP4, NVFP4, MXFP6, plus INT8/FP8 activations) for both dense and MoE layers. Marlin is an SM80-era kernel: it has no architecture-specific paths for Hopper or Blackwell,...
Windows: `cmake -G Ninja` builds the nested llama-server with the Visual Studio generator (CUDA install fails, CPU payload ships without llama-server.exe)
...only | install fails: `file INSTALL cannot find ...\bin\Release\ggml-cuda.dll` | Workarounds: set `CMAKE_GENERATOR=Ninja` in the environment instead of passing `-G`, or add `-DCMAKE_CONFIGURATION_TYPES=Release` to the outer configure. A separate, smaller thing I noticed while...
CUDA: on Volta, FA dispatch for DKQ=576 picks a 16-column MMA kernel (no sm_70 code) when gqa_ratio % 16 == 8
### Summary On Volta (sm_70) the CUDA flash-attention dispatch can launch `flash_attn_ext_f16` with `ncols1*ncols2 = 16`. That variant compiles to `NO_DEVICE_CODE` on Volta (`fattn-mma-f16.cuh`: `#ifdef VOLTA_MMA_AVAILABLE if (ncols1*ncols2 `. ### Mechanism (DKQ=576 / DV=512,...
Linux/NVIDIA: `heap_budgets` re-enumerates Vulkan physical devices on every step, about 17 ms CPU per decoded token
...about 6.4 ms, and they sit inside about 18.8 ms of wall time. The executor thread is the bottleneck. ### Where the time goes I sampled the `magnitude-execu` thread's stack with gdb during one request: 425 samples, all in decode. 96.7% of them (411) were inside...
Feature Request: MXFP4 on CUDA Blackwell could use MXFP8 activations via the mxf8f6f4 block-scaled MMA
### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...
clef-flash: `Clef: non-finite logit` (HTTP 500) on /v1/systemone — Windows, reproduces on CUDA and CPU-only, also after full reinstall
### What is the issue? [server.log](https://github.com/user-attachments/files/33094056/server.log) Calling `/v1/systemone` with `clef-flash` always fails with HTTP 500: {"error":"{\"error\":{\"code\":500,\"message\":\"Clef: non-finite logit\",\"type\":\"server_error\"}}"} The...
llama-server segfault in ggml_gallocr_alloc_graph during clip_encode (qwen3-vl:8b) with a second model loaded; CUDA 'resource allocation failed'
### What is the issue? `llama-server` crashes (SIGSEGV, NULL read) while encoding an image for `qwen3-vl:8b` when another large model is already loaded on the same two GPUs. Within 45 seconds it crashed three times; ollama then restarted the runner and the same request worked....
[Bug]: --moe-backend b12x on SM121 (DGX Spark): illegal memory access in CUDA-graph capture with padded rows, worker crash on prefill
_Posted by Claude (AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work._ ### `--moe-backend b12x` (native b12x NVFP4 MoE) on SM121: illegal memory access with padded graph batches, then a worker crash on prefill **Setup** - vLLM 0.30.0, b12x...
Feature Request: CUDA: reduce VRAM usage of FA convert buffer for quantized KV caches
### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...
0.2.3 selects the AMD iGPU as primary device on hybrid (dGPU + iGPU) machines; dGPU VRAM excluded from model fit
# 0.2.3 selects the AMD iGPU as primary compute device on hybrid (dGPU + iGPU) machines - dGPU VRAM excluded from catalog fit, catalog shrinks 20 → 14 models ## Summary After updating from **0.1.5** to **0.2.3** on a hybrid machine (NVIDIA RTX 5070 dGPU + AMD Radeon integrated...
Vulkan iGPU (AMD Ryzen 7700, Radeon Graphics) auto-selected despite manifest backend=cpu; all model loads fail numerical validation, no way to force CPU
## Summary On a Windows machine with an **AMD Ryzen 7 7700 (integrated Radeon graphics, no dGPU)**, the shipped `icn-base-windows-x64-msvc` artifact is marked `"backend": "cpu"` in the release manifest, but the binary reports `"backends": ["cpu","cuda","vulkan"]` and automatic...
Windows auto-update can leave cuda_v12\ggml-cuda.dll as .tmp, causing 0 B VRAM / CPU fallback
...the update (or automatically repair) if `ggml-cuda.dll` is missing; - surface a clearer error when CUDA runtime DLLs are present but the CUDA backend DLL itself is absent. Possibly related: #12618, but this report identifies a specific missing backend file and a same-size...
Windows CUDA: llama-server --list-devices intermittently returns empty stdout with exit code 0 while GPU discovery succeeds
...which narrowly scoped additional evidence is useful. Exact internal cause unknown. No workaround or fix is asserted. Related but not established duplicates: [Ollama #17375](https://github.com/ollama/ollama/issues/17375) concerns a nonzero access-violation exit; [llama.cpp...
MLX compile-cache CHECK failed spams logs on non-MLX hardware (Windows, no CUDA/Apple Silicon)
On every `ollama` command (including `ollama list`, `ollama serve`, `ollama run`), the following error is printed to the console before the actual output, regardless of which command was run: Sep 3 2026 16:31:11 - ERROR - generated.c:2650 - CHECK failed: mlx_compile_cache_new_...
Companies and products named
About this problem
Evidence
17 posts from 17 people in 4 places, about 2 a week over 66 days. Mostly on ollama/ollama, ggml-org/llama.cpp, magnitudedev/magnitude. Tools named alongside: Ollama, GitHub, llama.cpp, vLLM.
Frustration Frustration 1 of 3· Seen on GitHub issues
How it was grouped
Posts that state a pain and match the "local-llm" rule about NVIDIA. First post Sep 7, 2026, latest Nov 11, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running llama.cpp locally hits memory and speed limits
54 posts from 50 people on GitHub issues, Reddit, Stack Overflow.
Running Ollama locally hits memory and speed limits
48 posts from 46 people on GitHub issues, Reddit.
Running vLLM locally hits memory and speed limits
27 posts from 26 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
24 posts from 23 people on GitHub issues, Reddit, Hacker News.
Running Qwen locally hits memory and speed limits
20 posts from 20 people on GitHub issues, Reddit, Stack Overflow.
Running Hugging Face locally hits memory and speed limits
19 posts from 16 people on GitHub issues, Reddit.
Running OpenAI locally hits memory and speed limits
14 posts from 14 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
11 posts from 11 people on GitHub issues, Hacker News.
Running PyTorch locally hits memory and speed limits
10 posts from 6 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other NVIDIA problems
Trouble with NVIDIA phone
Top post: "Unable to verify NVIDIA account ,BD phone number not supported", one of 12 posts from 12 people on Community forums.
NVIDIA vulnerabilities are hard to triage and patch
77 posts from 77 people on Glyph Ibis.
NVIDIA users keep filing bugs
Top post: "`clef-flash` decision model always fails on `/v1/systemone`, Clef: non-finite logit (CUDA) / Clef: cannot", one of 44 posts from 41 people on GitHub issues, Community forums, Reddit.
NVIDIA GPU access and drivers are painful
13 posts from 13 people on GitHub issues, Mastodon, Reddit.
NVIDIA upgrades break working setups
11 posts from 11 people on GitHub issues.
Trouble with NVIDIA account
Top post: "NVIDIA Careers Workday Account Locked - Unable to Complete Employee Referral Application", one of 6 posts from 5 people on Community forums.
NVIDIA lock-in makes switching costly
5 posts from 5 people on GitHub issues, Hacker News, Bluesky.
NVIDIA runs out of memory
5 posts from 5 people on GitHub issues.
NVIDIA error messages are cryptic
4 posts from 4 people on GitHub issues, Reddit.
Trouble with NVIDIA slower
Top post: "CUDA training loop runs ~50x slower while a DataLoader iterates in a background thread, since 2.10.0", one of 4 posts from 4 people on GitHub issues.
Fine-tuning NVIDIA is costly and unpredictable
3 posts from 3 people on GitHub issues.
Trouble with NVIDIA process
Top post: "`HuggingFaceEmbeddings(multi_process=True)` starts CPU workers on non-CUDA accelerators", one of 3 posts from 3 people on GitHub issues.
History