Problemsby GlyphRunning OpenAI locally hits memory and speed limits
13 posts from 13 people12d activeSteady, +2 this week
Posts per day
The posts behind it
server: /v1/realtime audio output ignores output language/voice (non-English speech gets English accent; llama-tts with --tts-lang works)
### Name and Version llama-server 0.8.0 (b11960-8642541b2), Homebrew, macOS arm64 (Apple M2). Code path also unchanged on master as of 2026-11-04 (v0.9.0). ### Operating systems Mac ### Which llama.cpp modules do you know to be affected? llama-server ### Problem description &...
feat: Add MiniCPM-OCR as a content extraction engine
...`Text Recognition:`, `Table Recognition:`, `Formula Recognition:` ### Alternatives or Workarounds Run MiniCPM-OCR outside Open WebUI, then upload the extracted text as a file. This works, but it needs a separate script and does not use the Documents settings.
Saving a gpt-oss loaded with `Mxfp4Config(dequantize=True)` loses the experts
### System Info - `transformers` 5.22.1, and main at 676859711989dd9178fbbe8968e5a1f86d519712 - `torch` 2.9.1, H100 - 4.56.2 saves correctly ### Who can help? @SunMarc @MekkCyber ### Information - [ ] The official example scripts - [x] My own modified scripts ### Tasks - [ ] An...
Concurrent chat completions trigger "target state capacity changed after availability check", which unloads the model until the ICN restarts (0.2.13)
...## How often It happened once. The same scenario ran at least twice more without failing, so it looks timing-dependent. We have not yet reproduced it outside our app. ## Possible cause (our guess) The message points at page sharing between histories. The two chat requests shared...
[Bug]: Gemma 4 + online --quantization fp8: image requests crash since v0.23.0 (FP8-cast pixel values reach quantized patch_embedder.input_proj); v0.22.0 works
...from an earlier bisect that used the same server arguments.) **Where it fails.** The failing layer is the vision tower's patch embedder (`vt.patch_embedder` → transformers `Gemma4VisionPatchEmbedder.forward` → `self.input_proj(pixel_values)`), and `input_proj` has been replaced...
server: /v1/responses reports a truncated tool call as completed and never emits function_call_arguments.done
## Summary Two deviations from the OpenAI Responses streaming contract in the `/v1/responses` endpoint: 1. **`response.function_call_arguments.done` is never emitted.** Only `response.function_call_arguments.delta` exists. The arguments are delivered, then the item is closed...
[Bug]: Qwen3.8-Flash-Next checkpoints with quantized q/k/v and an unquantized QSA indexer fail to load since #59533 (v0.32.0)
### Your current environment The output of python collect_env.py Collected with `python -m vllm.collect_env` inside the image where the failure happened (`vllm/vllm-openai:nightly-43b4aaea3`, 4x A100-PCIE-40GB). The code involved is unchanged in v0.32.0 and in `main` at 371745e....
openai: reasoning_effort "xhigh" is rewritten to "max", breaking Qwen3.8 chat templates (HTTP 500)
...of this rewrite issue - #18121 - another recent report of OpenAI-path effort handling ## Workaround for anyone hitting this Omit `reasoning_effort` entirely (defaults to `xhigh` in these templates), or send `low` / `medium`. A manifest that selects the built-in renderer also...
Qwen3.6 35B-A3B: OMP task outputSchema triggers grammar binding error (Too many items, limit 50000)
...error would be more useful than attributing the failure to `messages`. ### Temporary workaround This has been tested to return `OK`. It limits available tools and is not a complete fix. No model downgrade or persistent configuration change was needed.
[Bug]: Qwen3.8-Flash-Next AutoRound/INC checkpoint fails because PLE rejects INCConfig even though PLE is unquantized
...Model dtype: `bfloat16` - Quantization detected by vLLM: `inc` - Max model length in the failing deployment: `262144` The same checkpoint works with the dedicated Qwen3.8 preview image: ### 🐛 Describe the bug `Intel/Qwen3.8-Flash-Next-W4A16-AutoRound` fails during model...
CPU-only x86-64 Linux (Intel N100): first inference never completes — worker spins forever in seismic X86V3 dot during lazy model load
## Summary Magnitude serves the model and reports it ready, but the **first** chat completion never returns on CPU-only Intel N100 hardware. The inference worker spins at 200-390% CPU indefinitely (one run observed for 27 minutes) and `kernels/artifacts` stays empty for the...
Support Prism ternary GGUFs (PQ2_0 type 142 / PTQ1_0 type 143): import fails with unsupported tensor size overflows
...and then decodes garbage. Worth knowing for anyone who tries the "portable" pack as a workaround. ## What it would take 1. llama.cpp: kernels for types 142/143 plus the `prism.hadamard.*` activation transform, tracked in ggml-org/llama.cpp#29058 (the metadata contract, block...
RFC: image, video and audio generation from diffusion GGUFs (LTX-2)
### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...
Companies and products named
About this problem
Evidence
13 posts from 13 people in 6 places, about 1 a week over 59 days. Mostly on ggml-org/llama.cpp, vllm-project/vllm, magnitudedev/magnitude. Tools named alongside: NVIDIA, llama.cpp, Hugging Face, GitHub.
Frustration Frustration 1 of 3· Seen on GitHub issues
How it was grouped
Posts that state a pain and match the "local-llm" rule about OpenAI. First post Sep 7, 2026, latest Nov 4, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running Ollama locally hits memory and speed limits
47 posts from 45 people on GitHub issues, Reddit.
Running llama.cpp locally hits memory and speed limits
45 posts from 41 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
22 posts from 21 people on GitHub issues, Reddit, Hacker News.
Running vLLM locally hits memory and speed limits
21 posts from 20 people on GitHub issues, Reddit.
Running Qwen locally hits memory and speed limits
18 posts from 18 people on GitHub issues, Reddit, Stack Overflow.
Running Hugging Face locally hits memory and speed limits
17 posts from 15 people on GitHub issues, Reddit.
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
8 posts from 8 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running PyTorch locally hits memory and speed limits
5 posts from 5 people on GitHub issues, Hacker News.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other OpenAI problems
Trouble with OpenAI responses
Top post: "completion_cost() raises ModelNotMappedError for groq/openai/gpt-oss-* responses, even when model=groq/... is", one of 18 posts from 16 people on GitHub issues, Community forums.
OpenAI runs out of context on real work
8 posts from 8 people on GitHub issues, Community forums.
OpenAI fails to return valid structured output
12 posts from 12 people on GitHub issues, Hacker News.
OpenAI users keep asking for help
Top post: "Multi-file uploads failing and ZIP archives not parsing / sandbox timeout (100% Limits Remaining)", one of 49 posts from 47 people on Community forums, GitHub issues, Reddit.
Trouble with OpenAI assistant
Top post: "Bedrock Converse forwards OpenAI-style content parts in user and assistant messages unconverted", one of 3 posts from 3 people on GitHub issues.
OpenAI rate limits and usage caps block work
6 posts from 6 people on GitHub issues, Community forums.
Trouble with OpenAI calls
Top post: "Unable to connect to dot calls across mobile, web and desktop", one of 7 posts from 6 people on GitHub issues, Community forums.
OpenAI upgrades break working setups
25 posts from 25 people on GitHub issues, Reddit, Community forums.
Trouble with OpenAI chat
Top post: "Chat attachments inside Projects intermittently fail to materialize into /mnt/data, while File Library and", one of 12 posts from 12 people on Community forums, GitHub issues, Reddit.
Trouble with OpenAI image
Top post: "Image generation is broken for serious work", one of 8 posts from 8 people on GitHub issues, Community forums, Reddit.
Trouble with OpenAI decisions
Top post: "OpenAI Decisions API", one of 6 posts from 4 people on GitHub issues.
Trouble with OpenAI reasoning
Top post: "glm-5.3 can enter endless reasoning and eventually abort tasks in both OpenCode and ZCode, while official", one of 5 posts from 5 people on GitHub issues.
History