Problemsby GlyphRunning Ollama locally hits memory and speed limits
47 posts from 45 people31d activeRising, +8 this week
Posts per day
The posts behind it
Vulkan: GLM-OCR vision encoder produces garbage on Intel Arc A380 for full-page images (works on CPU and SYCL)
### Name and Version /app/llama-mtmd-cli ghcr.io/ggml-org/llama.cpp:full-vulkan --version version: 0.9.0-dev (build 12169, commit 3ed7284f2) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan ### Hardware Vulkan0: Intel(R) Arc(tm) A380...
MLX support for dense Nemotron / Luciole Safetensors models
Loading a direct Safetensors-to-MLX conversion of OpenLLM-France/Luciole-1B-Instruct-1.1 on Ollama v0.40.15 fails with `unsupported architecture: NemotronForCausalLM`. The MLX runner registers Nemotron-H, but not the dense Nemotron architecture used by Luciole 1B and 23B. I...
Support Hindsight's decision model, hs-decide-1-flash
I work at Vectorize on [Hindsight](https://github.com/vectorize-io/hindsight), an open-source memory system for agents. We recently released our own decision model, [hs-decide-1-flash](https://huggingface.co/hindsight-ai/hs-decide-1-flash), which handles Hindsight's core memory...
mlxrunner: request stop sequences are ignored
### What is the issue? Models served by the MLX runner ignore request stop sequences. `Options.Stop` reaches `mlxrunner` but the decode loop in `mlxrunner/pipeline.go` never checks it, so generation continues to EOS or `num_predict`. Reproduction on 0.40.14 (official macOS arm64...
pull: verify GGUF blobs during download to avoid post-pull re-hash latency
...that forces a **single sequential stream** and disables resume as a digest-mismatch workaround. Different goal and trade-offs; this proposal keeps parallelism and is about default pull latency. - **#15028**: pre-rename re-read verification / HTTP status hardening (reliability;...
[Bug] Qwen2.5-Coder and Qwen3 fail native tool calling on Ollama 0.40.13
### What is the issue? I am building a private local AI assistant called **Maistos Stack Hub**, using Ollama, Open WebUI, and a Docker-based Python execution sandbox on Windows 11. I am experiencing native tool-calling failures with two models: - **`qwen2.5-coder:7b`** generates...
Ollama GitHub Issues
...path (CUDA + CLIP), not the model, quantization, prompt, or image encoding. **Workaround in place:** derived model with `PARAMETER num_gpu 0`. ## 5. Suspected area - Vision encoder (CLIP) CUDA kernels produce degenerate activations on this GPU/driver combination, leading to...
Ollama GitHub Issues
...path (CUDA + CLIP), not the model, quantization, prompt, or image encoding. **Workaround in place:** derived model with `PARAMETER num_gpu 0`. ## 5. Suspected area - Vision encoder (CLIP) CUDA kernels produce degenerate activations on this GPU/driver combination, leading to...
VRAM prediction over-counts KV cache 4-8x for hybrid models (recurrent/SSM blocks, OLLAMA_KV_CACHE_TYPE ignored), evicting runners loaded with keep_alive=-1
### What is the issue? `PredictServerVRAM` (`llm/llama_server.go`) over-predicts the KV cache by **4-8x** for hybrid architectures (recurrent/SSM blocks, e.g. Qwen3.5/3.8, Qwen3-Next-style models), and always bills it as f16 regardless of `OLLAMA_KV_CACHE_TYPE`. Combined with...
Qwen3.8-27B hybrid-attention GGUFs fail to load: qwen3next: layer 64 missing attn_qkv/attn_gate projections
...(Qwen3.8-27B, hybrid-attention / MTP architecture) fail to initialize in Ollama: The **same file loads and generates correctly with stock llama.cpp** (v0.38.x, build `b11960-8642541b2`), so the GGUF tensors appear intact - the Ollama runner appears to lack support for the...
Memory estimate for Gemma 4 E-series (gemma4:e4b) is 3-4x actual VRAM, causing needless evictions
### What is the issue? The scheduler's memory prediction for `gemma4:e4b` is 3-4x the VRAM the model actually uses, so it evicts other models that would fit next to it. It looks like the CPU-mapped per-layer embedding weights of the Gemma 4 E-series are counted as GPU memory....
ollama show --modelfile writes a Modelfile that ollama create rejects when a SYSTEM/TEMPLATE/LICENSE/MESSAGE value contains """ followed by whitespace
### What is the issue? `ollama show --modelfile` serializes values by wrapping them in `"""…"""` with no escaping. If the stored value itself contains `"""` followed by a space or newline, the output is unparseable. On re-parse, the value ends at the inner `"""` and the rest is...
error loading model: unknown model architecture: 'lfm3' [v0.40.12]
### What is the issue? When attempting to run the Hugging Face LiquidAI LFM3-1.7B GGUF model via Ollama, the process fails with > error loading model: unknown model architecture: 'lfm3' even though I'm on version 0.40.12. When pulling it from the Ollama registry, it says: >...
/api/chat intermittently fails with 500 "llama-server apply-template error … write: broken pipe" right after a long reply (0.40.8 and 0.40.11, macOS)
...- 23:56:38 | 500 | 3.18s | ::1 | POST "/api/chat" Pattern (from server.log) The failing call always comes 3-4 seconds after a long generation finished with a 200, e.g. 23:56:34 | 200 | 53.57s | POST /api/chat, then 23:56:38 | 500 | 3.18s. Preceding replies were 15-50s long...
Proposal: native HarmonyOS PC support, starting with CPU inference
## Problem and motivation HarmonyOS PC users currently lack an officially supported native Ollama distribution. Supporting the platform would let users run local models through Ollama’s existing CLI, HTTP API, and desktop experience. The work extends beyond compiling the...
/v1/systemone: GGUF decision models without a decision head are refused since 0.40.4; support models that score answer letters from an ordinary language-model head
## Summary Since 0.40.4 (it works on 0.40.3), `/v1/systemone` returns HTTP 400 for a GGUF model that declares `CAPABILITY decision` in its Modelfile but has no `decision.type` key: Pulling again does not help: the published file is what is refused. From the source...
Free-memory refresh (3 s deadline) times out on a 6-GPU host around model loads → stale VRAM → spurious eviction (incl. a CPU-only model)
### What is the issue? On a 6-GPU Linux host, the free-memory refresh that runs before each model load regularly hits its 3-second deadline: The scheduler then decides on stale numbers. Typical sequence when we unload two large models and load them back in order (big MoE first):...
flint:40b-a2b-gguf-nvfp4 MLX is slower than flint:40b-a2b-qat-int4 on Apple silicon
### What is the issue? Throughput for the following models is confusing, then model marked as MLX is significantly slower than non marked model. See tokens/s on MacBook Pro M3 36Gb (prompt: What can you do?): flint:40b-a2b-gguf-nvfp4 (MLX) flint:40b-a2b-qat-int4 ### Relevant log...
Scheduler stalls: runner marked "expire immediately" for eviction never unloads while other clients keep using it (keep_alive overrides the eviction)
...→ `resetting model to expire immediately` → the busy runner never unloads. ## Workaround Explicitly asking Ollama to unload the blocking model unblocks the scheduler within seconds: (Our chat client now does this automatically when the first token doesn't arrive within 20 s and...
Plz add Ollama set nothink in the UI
Been an ollama user for long, but the UI is lagging behind other AI model competitors. Please add /set nothink within the UI so that qwen and other models stop thinking infinitely bruh, its wasting tokens for the same output. Thanks! Hope we can also get some UI upgrade, coz...
Model thinks forever.
...its mind and never reaching a final result. Do you know how I can fix this? It's already annoying when it takes a long time cause I have the RTX 5060 with 8 GB of VRAM, which is limiting for the model. And all the other models seem stupid, so this was the best I could find....
Which models can I realistically run on a 4GB laptop vs a 12GB PC?
Hey everyone,. I am looking to get into local models but need some advice on my hardware. I have two different machines available and want to know what my realistic options are for both.. My laptop has an RTX 3050 4GB VRAM, an i5 12th gen CPU, and 16GB of DDR4 RAM.. My PC has an...
D1 vision: repeated /v1/systemone HTTP 500 with memory-slot/KV-cache errors after sequential requests
...prevent recurrence. I do not yet have a standalone minimal reproduction or a shareable failing image. Expected: sequential image decision requests continue to succeed, or return an actionable error if an input/resource limit is reached. Actual: repeated image-processing HTTP 500...
Ollama "official" VSCode extension issues
...) has some issues. Has anyone found workarounds for these?. It tells me to increase token size settings to 32K in ollama when I already have it at 32K in the ollama settings. My GPU is a Geforce 5070 with 12G VRAM so 32K might be more capacity than supported.. . Only gemma4...
Qwen3.5 4B returns thinking-only responses for a long conversation on Ollama 0.40.2
### What is the issue? ## Observed behavior The installed stock `qwen3.5:4b`, with request-local `options.num_ctx:65536`, returns HTTP 200 and completes with `done:true`, `done_reason:"stop"`, and positive `eval_count`, but only `message.thinking`: no content or tool calls. The...
ollama list / GET /api/tags show a duplicate model and a bogus llamacpp:<sha> tag after the local compat GGUF migration
...name works, so it's a listing/manifest bug, not a functional embedding failure. Workaround: ollama rm llamacpp: clears the extra row but it's recreated on next load; clients can defensively de-duplicate by name and hide names matching ^llamacpp:[0-9a-f]{64}$. ### Relevant log...
How are you all managing multiple GPUs and models across your homelab? My setup is becoming a mess
I have been running local LLMs in my homelab for a while now and I am hitting a wall with how fragmented everything is.. Right now I have LM Studio on a Ryzen 9 9950X with an RX 7900 XTX, another LM Studio on an i5-10400 with an RTX 3050, and Ollama on my MacBook Pro M3 - and...
Windows: model unusable after 0.40.0 migration writes its manifest as a symlink ("untrusted mount point")
### What is the issue? After Ollama auto-updated from 0.35.1 to 0.40.0, qwen3.5:9b stopped working. Every request for it fails, in the CLI and over the API: > ollama show qwen3.5:9b Error: CreateFile C:\Users\Dave\.ollama\models\manifests-v2\ollama.com\library\qwen3.5\9b: The...
MLX: quantized decision models are slower than bf16 at prefill (M5 Pro)
### What is the issue? I'm building a Snake game played in real time by local decision models, using Ollama's /v1/systemone endpoint. Each move has to arrive within the game's tick, so I was looking for the fastest format of each model. On 0.40.0, the bf16 MLX tags answer faster...
gemma4: a Gemma 4 12B GGUF created under a name without "12b" gets the gemma4-small renderer (11.9B is under the 12.0B threshold)
...a template it was not given by Google, and whether it is depends on the model's name. **Workaround.** Name the renderer in the Modelfile: That is what I ship on `aminroudaki/decisio-gemma`. **Possible fixes**, in the order I would pick them: - Prefer the GGUF's...
MLX runner: bf16 Gemma 4 decodes at ~1 tok/s on M2 Ultra; GPU idle ~96% of each step, runner waits in command-buffer submit
### What is the issue? `gemma4:26b-mlx-bf16` and `gemma4:31b-mlx-bf16` decode at 0.8-1.6 tokens/s on an M2 Ultra (192 GB). Prompt processing is normal. The 4-bit MLX build and the GGUF builds of the same models run at normal speed on the same machine. Same prompt for every row...
Bug: "unsupported tensor size overflows" when importing GSQ-RCO quantized Qwen3.8-Flash-Next, despite qwen4exp architecture being supported
...a default `block_size` of 256 for an unrecognized or custom quantization enum, or fails to read the custom block size from the GGUF metadata, the math will break down and result in an invalid/overflowing size calculation. ### Suggested Fix - Audit the GGUF tensor parsing logic...
mlx: per-layer quantization overrides ignored on import — model fails to load with quantized_matmul shape mismatch
### What is the issue? Importing an MLX safetensors checkpoint that uses **mixed precision** - 4-bit weights with per-layer 8-bit overrides - succeeds, but the model cannot load. The runner applies the global `bits` value to tensors whose scales were built for the override, and...
I wanted a native iOS client for Ollama without subscriptions or cloud relays, so I built Eron. v1.3 just added Workspaces and zero-lag streaming.
...you, I run models locally on my Mac/homelab, but using web UIs on an iPhone was always frustrating (safari timeouts, clunky scrolling, no system integrations).. I built Eron nearly 8 months ago as a clean, native iOS companion. You point it directly at your Ollama IP/URL (via...
think: "low"/"medium"/"high" ignored for Qwen3.8 GGUF whose chat template supports reasoning_effort (always runs at template default xhigh)
### What is the issue? For a Qwen3.8 27B GGUF whose embedded chat template supports graded reasoning effort, `/api/chat` accepts `think: "low"`, `"medium"` and `"high"` but behaves the same as `think: true`. Only `think: false` changes anything. The template in the GGUF...
Ollama forcefully unload models instead of load concurrent models in RAM
...max loaded models parameter set to 2 and the parallel parameter is set to 2. Why is it not working? And how do I make it to work?. EDIT. I think I solved the issue: Basically I need to wrap the existing model in a new model descriptor passing the "PARAMETER num_gpu 0" in a .mf...
Rocm GPU VRAM ignored when evicting models
### What is the issue? probably a duplicate of #16462 [ollama_vram_bug.txt](https://github.com/user-attachments/files/32973747/ollama_vram_bug.txt) When trying to load multiple models only a part of the available (unified) VRAM is used and 'older' models are evicted even though...
Decision models: a basal-1.0 encoding and per-type temperatures for /v1/systemone
## Problem `/v1/systemone` renders every question in one prompt format (or `clef`), but decision models are trained on one prompt format each. [basal-1.0](https://github.com/rkinas/basal) (Apache-2.0; 1.5B and 4.5B typed-decision models specialised in Polish, also English; on...
[LLM-jp-4] parser for harmony output with a space after special tokens
LLM-jp-4.1 (https://huggingface.co/llm-jp/llm-jp-4.1-8b-thinking-gguf, also 32b-a3b and 33b) uses the gpt-oss harmony format, but its tokenizer decodes a space after every special token: The `harmony` parser does not recognize ` analysis` as a header, so everything ends up in...
Model support: Bongard (T5Gemma2) for /v1/systemone
I would like to contribute native Bongard support through the existing `/v1/systemone` endpoint. Model: https://huggingface.co/AgentBull/bongard-mini Runtime and source: https://github.com/AgentBull/bongard Bongard-mini is a T5Gemma2 4B-4B encoder-decoder model with a trained...
Feature request: yield idle model VRAM under GPU-memory pressure
## Summary Please add an optional, best-effort policy for an idle Ollama model to release its VRAM early when another GPU application needs memory. ## Why The keep-alive timer is useful, but it does not account for a game, video editor, renderer, or other GPU-heavy application...
0xc0000005 access violation loading ANY model on Vulkan (AMD RX 6800 XT, driver 32.0.21045.5002)
## Summary Any model (tested with both a plain-transformer 7B and a hybrid SSM+transformer 27B) crashes `llama-server` with `exit status 0xc0000005` (access violation) when loaded on the Vulkan GPU backend. CPU inference works fine. This looks like the same underlying issue as...
Feature Request: Support Prism PQ2_0 (GGML type 142) and PTQ1_0 (type 143) used by Ternary-Bonsai-2
### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...
Add MLX support for Bonsai's low-bit (1-bit/2-bit) quantized weights to the new 0.19 MLX backend.
Add MLX support for Bonsai's low-bit (1-bit/2-bit) quantized weights to the new 0.19 MLX backend. The 2-bit ternary format already runs on stock MLX, so this may only require letting ollama pull fetch arbitrary MLX repos rather than registry-only models. The 1-bit format needs...
Multi-GPU VRAM accounting uses discovery device names instead of child's log names
### What is the issue? ## Summary On a multi-GPU system, the scheduler's VRAM accounting diverges from actual because three lookup sites read device names assigned by discovery (e.g. `CUDA1`) while the maps they read from are keyed by the child llama-server's log names (e.g....
Does Ollama support IQ3_S quantization for Qwen3.8-27B-GSQ-RCO-GGUF? Returns empty content
## Question Does Ollama currently support running the `IQ3_S` quantization of `Qwen3.8-27B-GSQ-RCO-GGUF` from hf.co? ## Problem When running `hf.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S` via Ollama: - Generation completes (`done_reason: "stop"`) - `content` field is always...
llama-server malloc heap grows with request volume on macOS/Metal: 6.5 GB paged to swap while KV cache stays resident (0.32.15, Apple Silicon)
...the pipeline idle but the model still loaded, swapouts were **exactly 0** over 90 s. ### Workaround Evicting the model (`{"keep_alive": 0}`) between units of work terminates that `llama-server` process, and the heap goes with it. With eviction every source, same 8192 context and...
Companies and products named
About this problem
Evidence
47 posts from 45 people in 4 places, about 4 a week over 72 days. Mostly on ollama/ollama, r/ollama, ggml-org/llama.cpp. Tools named alongside: NVIDIA, llama.cpp, GitHub, Hugging Face.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit
Summary, model-written
Ollama local users face llama-server access violation crashes on the Vulkan GPU backend, parser decoding issues with special tokens, and model memory retention problems.
How it was grouped
Posts that state a pain and match the "local-llm" rule about Ollama. First post Aug 28, 2026, latest Nov 7, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running llama.cpp locally hits memory and speed limits
45 posts from 41 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
22 posts from 21 people on GitHub issues, Reddit, Hacker News.
Running vLLM locally hits memory and speed limits
21 posts from 20 people on GitHub issues, Reddit.
Running Qwen locally hits memory and speed limits
18 posts from 18 people on GitHub issues, Reddit, Stack Overflow.
Running Hugging Face locally hits memory and speed limits
17 posts from 15 people on GitHub issues, Reddit.
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
Running OpenAI locally hits memory and speed limits
13 posts from 13 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
8 posts from 8 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running PyTorch locally hits memory and speed limits
5 posts from 5 people on GitHub issues, Hacker News.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other Ollama problems
Ollama upgrades break working setups
21 posts from 17 people on GitHub issues.
Ollama error messages are cryptic
13 posts from 7 people on GitHub issues, Reddit.
Ollama runs out of context on real work
7 posts from 7 people on GitHub issues, Reddit.
Trouble with Ollama empty
Top post: "Ollama photo/label estimation returns empty JSON with reasoning models (gemma4, qwen3.6), request should send", one of 7 posts from 6 people on GitHub issues.
Serving models on Ollama is harder than it should be
6 posts from 6 people on GitHub issues, Reddit.
Trouble with Ollama mode
Top post: "auto-playback should read responses sentence by sentence while streaming, like call mode", one of 6 posts from 4 people on GitHub issues.
Trouble with Ollama pull
Top post: "Failed to pull `embeddinggemma-2:740m` on linux", one of 6 posts from 6 people on GitHub issues.
Trouble with Ollama local
Top post: "issue: MinerU local engine gets 401 from a MinerU 4.x server started with --api-key, even with the API key set", one of 4 posts from 4 people on GitHub issues.
Trouble with Ollama instead
Top post: "issue(a11y): pressing Enter on a notification's Dismiss button opens the chat instead of dismissing it", one of 4 posts from 3 people on GitHub issues.
Ollama fails to return valid structured output
4 posts from 3 people on GitHub issues.
Trouble with Ollama text
Top post: "Ollama Text Classifier modles slower under version 0.40.4 ?", one of 4 posts from 3 people on GitHub issues.
Trouble with Ollama crashes
Top post: "Vulkan backend: llama-server crashes (NTSTATUS 0xe06d7363) at model load on AMD Renoir iGPU (Ryzen 4700U,", one of 3 posts from 3 people on GitHub issues.
History