Running llama.cpp locally hits memory and speed limits

45 posts from 41 people26d activeRising, +8 this week

Posts per day45 posts, Aug 27 to Nov 9

The posts behind it

GitHub issuesggml-org/llama.cpp · birdnamooNov 8

llama-server closes the connection after every streaming (SSE) response while the response advertises "Keep-Alive: timeout=5, max=100" and no "Connection: close" — a client that reuses the connection sometimes fails with "the response ended prematurely" (b11977 and b12182, macOS arm64)

...close` (and omits `Keep-Alive`) so that clients do not reuse it. **Client-side workaround.** Send `Connection: close` on every request to the server; a loopback connection costs microseconds.

GitHub issuesggml-org/llama.cpp · 0wwafaNov 7

Feature Request: clover quantization fails with Failed to load model: This GGUF is missing the metadata key 'clover_moe.expert_shared_count'

### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...

1 replies

GitHub issuesggml-org/llama.cpp · wsj040303Nov 7

Misc. bug: backend sampling repeatedly reallocates pinned host compute buffers as requests finish

...The log confirms 29/29 layers on the GPU. No model/weight changes are made for the workaround below. Save the standalone client and exact fixture from the collapsed sections below. Once `/health` is ready, in another terminal: The client sends one extra warmup batch, then one...

GitHub issuesggml-org/llama.cpp · marcelromanoNov 6

Proposal: Quantized-KV GQA flash attention for AMD gfx906 (MI50)

**Status & Context:** Coordination proposal on existing ROCm flash-attention work; not a new issue or PR. Relevant upstream work: [PR #30792](https://github.com/ggml-org/llama.cpp/pull/30792) (quantized-KV tile attention on ROCm). This proposal offers complementary gfx906 target...

GitHub issuesggml-org/llama.cpp · marcelromanoNov 6

Proposal: gfx906 MMVQ decode and fused glue kernels

**Status & Context:** Technical discussion proposal for HIP/GFX optimization threads. Related upstream context: [#30344](https://github.com/ggml-org/llama.cpp/pull/30344) (HIP/GFX tuning) and [#30348](https://github.com/ggml-org/llama.cpp/pull/30348) (AMD register spill checks)....

GitHub issuesggml-org/llama.cpp · usrlocalbenNov 6

Feature Request: Kimi-K3.1(-Pro) model support

GitHub issuesggml-org/llama.cpp · DSmidgeNov 6

Misc. bug: SYCL and Flash Attention problem / warnings

...device SYCL0 but Flash Attention over K/V rows is assigned to device CPU (usually due to missing support) - resolve_fused_ops: Flash Attention over K/V rows not supported, set to disabled I tested with these two models: https://huggingface.co/lmstudio-community/Qwen3.8-27B-GGUF...

1 replies

GitHub issuesggml-org/llama.cpp · PaperSpectatorNov 3

Feature Request: Support custom special_eog_ids

1 reactions

GitHub issuesggml-org/llama.cpp · gstosNov 2

Vulkan: KHR_coopmat gives nondeterministic, wrong results on Intel Arc Pro B70 (BMG G31, Mesa 26.2.5) — encoder/decision model, F16 and Q8_0

...(not in the release tarball). If it helps, I can build it and report the failing MUL_MAT / FLASH_ATTN_EXT cases. - Nondeterminism suggests a race

10 replies

GitHub issuesggml-org/llama.cpp · giaoimgiaoNov 2

llama : add dense FFN support for Qwen3Next (Qwen3.5/3.8-27B dense) and fix SIGFPE when n_expert_used == 0

## Problem **Qwen3.8-27B** (a *dense* variant of the Qwen3-Next architecture) cannot be converted or loaded, even though `LLM_ARCH_QWEN3NEXT` and `build_qwen3next()` already handle most of it. Config summary (from `config.json`): Four separate blockers: 1....

1 replies

GitHub issuesggml-org/llama.cpp · YuWei-CHOct 27

Misc. bug: Op offload counts heads as batch size, creating isolated GPU splits in CPU layers

...causes the `MUL` to reach the offload threshold even during single-token decoding. The workaround at [llama-context.cpp:2872

GitHub issuesggml-org/llama.cpp · mengyaysOct 26

Feature Request: modifying the data layout of its K-quants format to improve GPU memory access efficiency

GitHub issuesollama/ollama · sunkapOct 25

Performance guidance: Qwen3.5 4B GGUF prompt processing on 16GB M4 (100% GPU, llama.cpp)

...recommended asking here with loaded-runtime readings and synthetic timing evidence. We need help establishing whether the prompt-processing performance below is expected and, if not, the smallest supported configuration/runtime correction. ### Environment - Apple M4 Mac mini, 16...

3 replies

GitHub issuesggml-org/llama.cpp · kossakmarcinOct 25

Feature Request: Dflash support for odd number cards (3x3060) with sm tensor - qwen3.8-27b

GitHub issuesggml-org/llama.cpp · KanisktenNOct 24

Add KailA architecture support

1 reactions

GitHub issuesggml-org/llama.cpp · Rocke1001fellerOct 24

Metal: bf16 MUL_MAT returns wrong results on pre-M5 integrated GPUs (Intel Iris, Common3 family) even with -ngl 0

...loops, short prompts look fine. This makes it look like a model/template bug and is very hard to diagnose. Passing `-ngl 0` does **not** help because GPU devices are still registered (fixed by #31090). Layer-wise bisection against the `transformers` eager reference (same...

1 replies

GitHub issuesggml-org/llama.cpp · mctylr-ghOct 20

Misc. bug: SYCL: SWIGLU_OAI aborts on f16 tensors

### Name and Version Base commit: 3ffb62935, build with GGML_SYCL_F16=ON, Release ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library) ### Command line ### Problem description & steps to reproduce The SYCL implementation of...

GitHub issuesggml-org/llama.cpp · mctylr-ghOct 18

Misc. bug: SYCL FLASH_ATTN_EXT failures

### Name and Version 01fc3aae8 (“model: add Beam (Reflection) (#30670)”) (2026-10-16) PR submitted for this issue: #30761 ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? libllama (core library) ### Command line #### case A (tensor stride for...

1 replies

GitHub issuesggml-org/llama.cpp · figkcoci-debugOct 16

vulkan full offload of a 27B TQ1_0 model crashes the Adreno 840 driver inside vkGetDeviceFaultInfoEXT

device Redmi K100 Pro Max, Snapdragon 8 Elite Gen 5 (SM8850), Adreno 840, Android 16 build fingerprint Redmi/songyuan/songyuan:16/BP2A.250605.031.A3/OS3.0.310.0.WGNCNXM vulkan driver build id 30e0772cdc77186f332f82a477169cb7 llama-bench built for arm64 android with...

GitHub issuesggml-org/llama.cpp · figkcoci-debugOct 16

quantized MUL_MAT gives wrong results on Adreno 840 while the same cases pass on Intel Arc

quantized MUL_MAT gives wrong results on Adreno 840 while the same cases pass on Intel Arc device Redmi K100 Pro Max, Snapdragon 8 Elite Gen 5 (SM8850), Adreno 840, Android 16 build fingerprint Redmi/songyuan/songyuan:16/BP2A.250605.031.A3/OS3.0.310.0.WGNCNXM vulkan driver build...

1 replies

GitHub issuesggml-org/llama.cpp · figkcoci-debugOct 16

test-backend-ops MUL_MAT tq1_0 fails on Vulkan on both Intel Arc and Adreno 840

...types pass, only tq1_0 fails the cpu reference is fine, only the vulkan kernels deviate failing case strings MUL_MAT(type_a=tq1_0,type_b=f32,m=16,n=2,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0,m_v=0,pad=0)...

1 replies

GitHub issuesggml-org/llama.cpp · BaloogaOct 16

Misc. bug: Vulkan: generation 2-4x slower with two GPUs since #29741 (GCN 1.0 / Tahiti, no ReBAR)

### Name and Version `v0.7.0` ### Operating systems Linux ### Which llama.cpp modules do you know to be affected? llama-server ### Command line ### Problem description & steps to reproduce # Vulkan: generation 2-4x slower with two GPUs since #29741 (GCN 1.0 / Tahiti, no ReBAR)...

3 replies

GitHub issuesggml-org/llama.cpp · SomeNobody21112Oct 15

Feature Request: gguf-py: add numpy quantisation for Q1_0 and Q2_0

1 replies

GitHub issuesollama/ollama · Kaniskten22Oct 10

proposal: add KailA (kail) architecture support to the llama.cpp compat layer

### Problem Ollama cannot load KailA models. KailA GGUF files declare `general.architecture = kail`, which no current build recognizes, so the loader fails at architecture resolution ("unknown model architecture: 'kail'") before any tensors are used. Users who hold KailA GGUFs...

1 replies

GitHub issuesmagnitudedev/magnitude · eliottthomas99Oct 10

Qwen3.8 27B (Q4) on M1 Max: every generation fails with sampling distribution is nonfinite (v0.2.6)

...(e.g. `attention_decode_k8v4`, `gated_delta_step_convolved`) or from the drafter. Is there a way to disable the drafter to isolate it? - The tuning cache is in `~/.magnitude/cache/kernels/tuning/` (76 entries, tuning identity `Apple M1 Max;metal;...;applegpu_g13s;...;os Version...

5 replies

GitHub issuesggml-org/llama.cpp · sadam266Oct 9

Misc. bug:

### Name and Version llama-server.exe --version version: 0.6.0-dev (build 11429, commit d81235049) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Other? (Please let us know in description), Windows ### Which llama.cpp modules do you know to be affected? Other...

GitHub issuesggml-org/llama.cpp · chrisjohnsonOct 6

qwen4exp QSA indexer OOMs once vision disables the scalar fast path (O(n_kv × ubatch) mask/score materialization) — the TODO in #29824 is this exact cost

## Bug: qwen4exp QSA indexer OOMs once vision disables the scalar fast path (O(n_kv × ubatch) mask/score materialization) - confirmed live, and the TODO in #29824 is this exact cost ### Summary Once any multimodal (M-RoPE 2D-positioned) image enters a long-lived KV cache,...

GitHub issuesggml-org/llama.cpp · Eu21kaOct 6

Misc. bug: Heap out-of-bounds read in /v1/embeddings when started with --embeddings --rerank (RANK pooling)

### Name and Version version: 0.6.0 (build 1, commit d812350) built with gcc (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0, x86_64-linux-gnu (ASan build: Debug + -fsanitize=address; release build also tested) ### Operating systems Linux ### Which llama.cpp modules do you know to be...

GitHub issuesggml-org/llama.cpp · heisejufengOct 3

HIP/ROCm: llama-server hangs at init (single-thread spin, no I/O) with --load-mode mmap or --cpu-moe on qwen4exp / Strix Halo (gfx1151)

### Name and Version - `version: 0.5.0-dev (build 703, commit 254b1773)` - self-built, HIP backend - `version: 0.5.0-dev (build 726, commit cb7934c5)` - self-built, HIP backend, includes #29825 - **identical behavior** Both hang the same way, so this is not fixed by current...

GitHub issuesggml-org/llama.cpp · mrBenBernOct 3

llama-quantize: GGML_ASSERT(size mismatch) when quantizing mmproj files containing 3D conv tensors (gemma4v audio conv1d)

...Every subsequent 3D tensor fails the same way (a.conv1d.1.weight [3,3,128,32], ...). Workaround (verified): keep 3D tensors in f32 via tensor-type overrides - llama-quantize --allow-requantize \ --tensor-type "a.conv1d.0.weight=f32" \ --tensor-type "a.conv1d.1.weight=f32" \...

GitHub issuesggml-org/llama.cpp · notforu777Oct 3

Misc. bug: llama-server with laya model aborts when the questions of a batch exceed n_ubatch

### Name and Version version: 0.5.0-dev (build 11374, commit b92761a51) built with GNU 11.4.0 for Linux x86_64 version: 0.5.0-dev (build 11364, commit 46ca246de) built with AppleClang 21.0.0.21000099 for Darwin arm64 ### Operating systems Linux, Mac ### Which llama.cpp modules...

1 replies

GitHub issuesggml-org/llama.cpp · yiwasuiOct 3

[Android/aarch64] Backend score prefers the SVE2 variant, which is 2x slower than the non-SVE variant on 128-bit VL mobile cores

...packaged in the APK; a first-class way to exclude them at packaging time would help. ### Workaround At the application level, calling `ggml_backend_load()` on a fixed preference order (`armv8.6_1` → `armv8.2_2` → `armv8.2_1` → `armv8.0_1`) instead of relying on...

GitHub issuesmagnitudedev/magnitude · devferrazZzOct 2

Qwen3.8 27B (Q4) fails to load on M1 Max: `attention_project_gemv` requests 512 threads per threadgroup, pipeline allows 448

...models status qwen3.8-27b:gguf:q4` reports the model as installed, but the runtime fails to load it: The model never loads, and `/inference/v1/models` lists it but cannot serve it. ### Expected The model loads. The catalog lists this model as "Comfortable (23 GB)" on this...

1 reactions1 replies

GitHub issuesggml-org/llama.cpp · Sahan-Chathumina-25Oct 2

Fix size-suffix threshold so exact 1M/1B/1T counts use the larger unit

### Description I noticed `model_weight_count_rounded_notation()` in `gguf-py/gguf/utility.py:22-30` picks the size suffix with strict `>` comparisons against `1e12`, `1e9`, and `1e6`. That means a model with exactly 1,000,000,000 parameters falls into the millions branch and...

GitHub issuesggml-org/llama.cpp · mirn0410Oct 2

Feature Request: Add support for RheoEcho ETET 1.0 24E 1.8B A1B Preview

GitHub issuesggml-org/llama.cpp · Martin-StanglOct 2

`common/fit`: MoE step 4 underflows `n_part` on the last device (guard tests `id` instead of `id_dense_start`) → near-endless fit loop

...(any binary that calls `common_fit_params`; also `llama-fit-params`) ## Command line The failing load, `gemma4` architecture: ## Problem description & steps to reproduce **Setup:** 9 × 6 GB CUDA devices (SM 8.6, 5800 MiB each). The model is an MoE GGUF with an mmproj, Q4_K_M....

1 replies

GitHub issuesggml-org/llama.cpp · pcuencaOct 1

Misc. bug: imatrix assertion failure in make_qkx3_quants

### Name and Version version: 0.5.0-dev (build 11253, commit 86ea01d05) built with AppleClang 21.0.0.21000101 for Darwin arm64 ### Operating systems Mac ### Which llama.cpp modules do you know to be affected? llama-quantize ### Command line ### Problem description & steps to...

GitHub issuesggml-org/llama.cpp · JuanqicaiSep 29

Misc. bug: windows: when the simple input reader reads EOF on stdin, all processes in the console are sent ctrl+c

### Name and Version version: 0.5.0-dev (build 11254, commit 8019dc563) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Windows ### Which llama.cpp modules do you know to be affected? llama-cli ### Command line ### Problem description & steps to reproduce When...

1 reactions4 replies

Redditr/LocalLLaMA · u/mrgreatheartSep 24

Epyc for inference

...at some point plus enough RAM to overflow bigger models than I can fit in VRAM.. I’m struggling to find clear information on whether this would actually be worth it for the cost.. I am currently able to run Qwen3.8-flash-next in two rather painful configurations (not using the...

GitHub issueshuggingface/transformers · imezxSep 16

Need Kimi K3 model architecture support

### Model description hey there, i saw transformers still havent support kimi k3 architecture yet, while llama.cpp for inference already. i have trained new tiny model from scratch using the kimi k3 architecture (0.5B & 1.35B), it also fully compatible with llama.cpp gguf. but...

1 reactions1 replies

GitHub issuesggml-org/llama.cpp · kof8855Sep 3

support Spark-X2.5-4B-GGUF

Failed to load model: This model is not supported yet. Try a different model. (Original error: llama.cpp does not support this GGUF's model architecture ('spark2_5'). The file is valid, but this model type cannot be run with llama-server.)

2 reactions1 replies

GitHub issuesggml-org/llama.cpp · maciejdzialoSep 1

bug: Qwen3.8 DFlash/MTP speculative emits OOB token id == n_vocab (248320) on Vulkan

## Summary On AMD Strix Point (`gfx1150`) Vulkan with llama.cpp **b10724**, Qwen3.8-27B + z-lab DFlash2 (or ggml MTP draft) fails decode with: **248320 equals `tokenizer.ggml.tokens` array length** - valid token ids are `0..248319`. This looks like a classic off-by-one in draft...

2 reactions6 replies

GitHub issuesggml-org/llama.cpp · John-194Aug 29

Feature Request: `--n-cpu-mode` FFN band selection for `--n-cpu-ffn`

6 reactions7 replies

GitHub issuesggml-org/llama.cpp · podolchakagencyAug 29

qwen4exp: slow decode on Pascal (sm_61) — 18 tok/s vs 29.5 for same-class qwen3.5moe with more active params

### Summary On 4× Tesla P40 (sm_61), `qwen4exp` (Qwen3.8-Flash-Next, 125B / 6B active) decodes at **~18 tok/s**, while `qwen3.5moe` (Qwen3.5-122B-A10B, 122B / **10B** active) on the same GPUs and the same build decodes at **~29.5 tok/s**. This is the opposite of what the...

2 reactions1 replies

GitHub issuesggml-org/llama.cpp · ranxiangleiAug 27

qwen4exp (PR #27742): multi-segment prompts degrade to '//////' on gfx1151 — deterministic repro

...usage already works great here (thinking mode included, after the chat-template workaround below). ### Environment - Strix Halo laptop APU: Ryzen AI MAX+ 395 + Radeon 8060S (`gfx1151`), ROCm 7.2, kernel 6.19.8 - Tested both #27774 head (`abdc7a0`) and #27742 head (`6c5afc86`) -...

2 reactions5 replies

Companies and products named

llama.cpp

45 posts

GitHub

16 posts

NVIDIA

8 posts

Hugging Face

7 posts

Ollama

2 posts

PyTorch

2 posts

About this problem

Evidence

45 posts from 41 people in 5 places, about 4 a week over 74 days. Mostly on ggml-org/llama.cpp, magnitudedev/magnitude, ollama/ollama. Tools named alongside: GitHub, NVIDIA, Hugging Face, Ollama.

Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit

Summary, model-written

Local llama.cpp users encounter decode failures due to invalid token array indices, unsupported GGUF model architectures, and unexpected decoding speed slowdowns on certain GPU setups.

How it was grouped

Posts that state a pain and match the "local-llm" rule about llama.cpp. First post Aug 27, 2026, latest Nov 8, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

The same problem elsewhere

Momentum went from fading to rising.

Nov 8, 2026

Momentum went from steady to fading.

Nov 1, 2026

Momentum went from rising to steady.

Oct 31, 2026

Summary written by a model: Local llama.cpp users encounter decode failures due to invalid token array indices, unsupported GGUF model architectures, and unexpected decoding speed slowdowns on certain GPU setups.

Oct 30, 2026

Momentum went from fading to rising.

Oct 30, 2026

Momentum went from steady to fading.

Oct 23, 2026

Momentum went from rising to steady.

Oct 22, 2026

Momentum went from fading to rising.

Oct 21, 2026

Momentum went from steady to fading.

Oct 10, 2026

Momentum went from rising to steady.

Oct 9, 2026

Momentum went from fading to rising.

Oct 5, 2026

Momentum went from steady to fading.

Oct 2, 2026