Data and AI struggles

2,114 problems · 377 companies

MLOps

Running NVIDIA locally hits memory and speed limits

17 posts from 17 people13d activeFading, +0 this week

Posts per day

Posts per day17 posts, Sep 7 to Nov 21

The posts behind it

GitHub issuesggml-org/llama.cpp · astroturfgamesNov 11

CUDA backend: fatal CUDA error + NVIDIA driver reset (nvlddmkm Event 153) during sustained batched prompt processing while an NVENC encoder shares the GPU — Vulkan build is stable at the same VRAM footprint (depth-dependent)

**Environment:** - NVIDIA GeForce RTX 4090 (24 GB), single GPU, driver 621.14 - Windows 26H2, build 26300 - tested at three servicing levels (Nov 2026 cumulative, Oct 2026 cumulative KB5127648, and the pre-October Sept 22 state); the crash reproduces at all three, only the...

1 replies
GitHub issuesggml-org/llama.cpp · idruskovNov 6

Feature request: make mixed-vendor (e.g. Intel Vulkan target + NVIDIA CUDA draft) speculative decoding actually worth using

### Context Setup: Intel Arc Pro B65 (32 GB, primary) + NVIDIA RTX 3050 (6 GB, secondary) on the same host. Goal: use the smaller NVIDIA card purely as a speculative-decoding draft device for a target model running on the Intel card via Vulkan. This works today, but only after...

GitHub issuesggml-org/llama.cpp · samir-lmsNov 5

Compile bug: CUDA11 sm_50 fails in lightning-indexer.cu

### Git commit e30ad80fd7fa09a858bf8b8f4118e6479809486e ### Operating systems Windows ### GGML backends CUDA ### Problem description & steps to reproduce The problem was produced on a clean Linux x86_64 CUDA 11.7 build which was targeting `sm_50`. It failed in...

1 reactions2 replies
GitHub issuesvllm-project/vllm · mgoinNov 4

[Tracking] Replace Marlin with Humming across NVIDIA architectures

### Motivation. Humming now covers the quantization formats Marlin serves (INT4/INT8 GPTQ/AWQ, FP8, MXFP8, MXFP4, NVFP4, MXFP6, plus INT8/FP8 activations) for both dense and MoE layers. Marlin is an SM80-era kernel: it has no architecture-specific paths for Hopper or Blackwell,...

2 reactions3 replies
GitHub issuesollama/ollama · KafKafrnZNov 3

Windows: `cmake -G Ninja` builds the nested llama-server with the Visual Studio generator (CUDA install fails, CPU payload ships without llama-server.exe)

...only | install fails: `file INSTALL cannot find ...\bin\Release\ggml-cuda.dll` | Workarounds: set `CMAKE_GENERATOR=Ninja` in the environment instead of passing `-G`, or add `-DCMAKE_CONFIGURATION_TYPES=Release` to the outer configure. A separate, smaller thing I noticed while...

GitHub issuesggml-org/llama.cpp · avpanasyukNov 2

CUDA: on Volta, FA dispatch for DKQ=576 picks a 16-column MMA kernel (no sm_70 code) when gqa_ratio % 16 == 8

### Summary On Volta (sm_70) the CUDA flash-attention dispatch can launch `flash_attn_ext_f16` with `ncols1*ncols2 = 16`. That variant compiles to `NO_DEVICE_CODE` on Volta (`fattn-mma-f16.cuh`: `#ifdef VOLTA_MMA_AVAILABLE if (ncols1*ncols2 `. ### Mechanism (DKQ=576 / DV=512,...

GitHub issuesmagnitudedev/magnitude · puhukOct 30

Linux/NVIDIA: `heap_budgets` re-enumerates Vulkan physical devices on every step, about 17 ms CPU per decoded token

...about 6.4 ms, and they sit inside about 18.8 ms of wall time. The executor thread is the bottleneck. ### Where the time goes I sampled the `magnitude-execu` thread's stack with gdb during one request: 425 samples, all in decode. 96.7% of them (411) were inside...

GitHub issuesggml-org/llama.cpp · timothyeburkeOct 17

Feature Request: MXFP4 on CUDA Blackwell could use MXFP8 activations via the mxf8f6f4 block-scaled MMA

### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...

GitHub issuesollama/ollama · lougomOct 6

clef-flash: `Clef: non-finite logit` (HTTP 500) on /v1/systemone — Windows, reproduces on CUDA and CPU-only, also after full reinstall

### What is the issue? [server.log](https://github.com/user-attachments/files/33094056/server.log) Calling `/v1/systemone` with `clef-flash` always fails with HTTP 500: {"error":"{\"error\":{\"code\":500,\"message\":\"Clef: non-finite logit\",\"type\":\"server_error\"}}"} The...

1 reactions6 replies
GitHub issuesollama/ollama · jmaxwell81Oct 6

llama-server segfault in ggml_gallocr_alloc_graph during clip_encode (qwen3-vl:8b) with a second model loaded; CUDA 'resource allocation failed'

### What is the issue? `llama-server` crashes (SIGSEGV, NULL read) while encoding an image for `qwen3-vl:8b` when another large model is already loaded on the same two GPUs. Within 45 seconds it crashed three times; ollama then restarted the runner and the same request worked....

GitHub issuesvllm-project/vllm · sethforprivacyOct 1

[Bug]: --moe-backend b12x on SM121 (DGX Spark): illegal memory access in CUDA-graph capture with padded rows, worker crash on prefill

_Posted by Claude (AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work._ ### `--moe-backend b12x` (native b12x NVFP4 MoE) on SM121: illegal memory access with padded graph batches, then a worker crash on prefill **Setup** - vLLM 0.30.0, b12x...

1 replies
GitHub issuesggml-org/llama.cpp · AbaiOct 1

Feature Request: CUDA: reduce VRAM usage of FA convert buffer for quantized KV caches

### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...

GitHub issuesmagnitudedev/magnitude · tsapenkodmitriiOct 1

0.2.3 selects the AMD iGPU as primary device on hybrid (dGPU + iGPU) machines; dGPU VRAM excluded from model fit

# 0.2.3 selects the AMD iGPU as primary compute device on hybrid (dGPU + iGPU) machines - dGPU VRAM excluded from catalog fit, catalog shrinks 20 → 14 models ## Summary After updating from **0.1.5** to **0.2.3** on a hybrid machine (NVIDIA RTX 5070 dGPU + AMD Radeon integrated...

1 replies
GitHub issuesmagnitudedev/magnitude · avsutorminOct 1

Vulkan iGPU (AMD Ryzen 7700, Radeon Graphics) auto-selected despite manifest backend=cpu; all model loads fail numerical validation, no way to force CPU

## Summary On a Windows machine with an **AMD Ryzen 7 7700 (integrated Radeon graphics, no dGPU)**, the shipped `icn-base-windows-x64-msvc` artifact is marked `"backend": "cpu"` in the release manifest, but the binary reports `"backends": ["cpu","cuda","vulkan"]` and automatic...

GitHub issuesollama/ollama · daleb0203Sep 30

Windows auto-update can leave cuda_v12\ggml-cuda.dll as .tmp, causing 0 B VRAM / CPU fallback

...the update (or automatically repair) if `ggml-cuda.dll` is missing; - surface a clearer error when CUDA runtime DLLs are present but the CUDA backend DLL itself is absent. Possibly related: #12618, but this report identifies a specific missing backend file and a same-size...

GitHub issuesollama/ollama · Algocrypt0777Sep 29

Windows CUDA: llama-server --list-devices intermittently returns empty stdout with exit code 0 while GPU discovery succeeds

...which narrowly scoped additional evidence is useful. Exact internal cause unknown. No workaround or fix is asserted. Related but not established duplicates: [Ollama #17375](https://github.com/ollama/ollama/issues/17375) concerns a nonzero access-violation exit; [llama.cpp...

GitHub issuesollama/ollama · znanovikSep 7

MLX compile-cache CHECK failed spams logs on non-MLX hardware (Windows, no CUDA/Apple Silicon)

On every `ollama` command (including `ollama list`, `ollama serve`, `ollama run`), the following error is printed to the console before the actual output, regardless of which command was run: Sep 3 2026 16:31:11 - ERROR - generated.c:2650 - CHECK failed: mlx_compile_cache_new_...

2 replies

Companies and products named

17 posts
7 posts
7 posts
7 posts
2 posts
1 post
About this problem

Evidence

17 posts from 17 people in 4 places, about 2 a week over 66 days. Mostly on ollama/ollama, ggml-org/llama.cpp, magnitudedev/magnitude. Tools named alongside: Ollama, GitHub, llama.cpp, vLLM.

Frustration Frustration 1 of 3· Seen on GitHub issues

How it was grouped

Posts that state a pain and match the "local-llm" rule about NVIDIA. First post Sep 7, 2026, latest Nov 11, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

The same problem elsewhere

Running llama.cpp locally hits memory and speed limits

54 posts from 50 people on GitHub issues, Reddit, Stack Overflow.

Running Ollama locally hits memory and speed limits

48 posts from 46 people on GitHub issues, Reddit.

Running vLLM locally hits memory and speed limits

27 posts from 26 people on GitHub issues, Reddit.

Running LLMs locally hits memory and speed limits

24 posts from 23 people on GitHub issues, Reddit, Hacker News.

Running Qwen locally hits memory and speed limits

20 posts from 20 people on GitHub issues, Reddit, Stack Overflow.

Running Hugging Face locally hits memory and speed limits

19 posts from 16 people on GitHub issues, Reddit.

Running OpenAI locally hits memory and speed limits

14 posts from 14 people on GitHub issues.

Running DeepSeek locally hits memory and speed limits

11 posts from 11 people on GitHub issues, Hacker News.

Running PyTorch locally hits memory and speed limits

10 posts from 6 people on GitHub issues, Hacker News.

Running Claude Code locally hits memory and speed limits

6 posts from 6 people on GitHub issues.

Running GitHub locally hits memory and speed limits

4 posts from 4 people on Reddit, GitHub issues.

Running Perplexity locally hits memory and speed limits

3 posts from 3 people on GitHub issues.

Other NVIDIA problems

Trouble with NVIDIA phone

Top post: "Unable to verify NVIDIA account ,BD phone number not supported", one of 12 posts from 12 people on Community forums.

NVIDIA vulnerabilities are hard to triage and patch

77 posts from 77 people on Glyph Ibis.

NVIDIA users keep filing bugs

Top post: "`clef-flash` decision model always fails on `/v1/systemone`, Clef: non-finite logit (CUDA) / Clef: cannot", one of 44 posts from 41 people on GitHub issues, Community forums, Reddit.

NVIDIA GPU access and drivers are painful

13 posts from 13 people on GitHub issues, Mastodon, Reddit.

NVIDIA upgrades break working setups

11 posts from 11 people on GitHub issues.

Trouble with NVIDIA account

Top post: "NVIDIA Careers Workday Account Locked - Unable to Complete Employee Referral Application", one of 6 posts from 5 people on Community forums.

NVIDIA lock-in makes switching costly

5 posts from 5 people on GitHub issues, Hacker News, Bluesky.

NVIDIA runs out of memory

5 posts from 5 people on GitHub issues.

NVIDIA error messages are cryptic

4 posts from 4 people on GitHub issues, Reddit.

Trouble with NVIDIA slower

Top post: "CUDA training loop runs ~50x slower while a DataLoader iterates in a background thread, since 2.10.0", one of 4 posts from 4 people on GitHub issues.

Fine-tuning NVIDIA is costly and unpredictable

3 posts from 3 people on GitHub issues.

Trouble with NVIDIA process

Top post: "`HuggingFaceEmbeddings(multi_process=True)` starts CPU workers on non-CUDA accelerators", one of 3 posts from 3 people on GitHub issues.

History

Momentum went from steady to fading.

Nov 11, 2026

Momentum went from rising to steady.

Nov 10, 2026

Momentum went from steady to rising.

Nov 8, 2026

Momentum went from fading to steady.

Nov 7, 2026

Momentum went from steady to fading.

Nov 6, 2026

Momentum went from fading to steady.

Nov 2, 2026

Momentum went from steady to fading.

Oct 24, 2026

Momentum went from fading to steady.

Oct 21, 2026

Momentum went from rising to fading.

Oct 8, 2026

Corroborated: 3 or more posts from 2 or more people or places.

Oct 5, 2026

Momentum went from steady to rising.

Oct 5, 2026

First seen with 2 posts.

Oct 4, 2026