Problemsby GlyphRunning LLMs locally hits memory and speed limits
22 posts from 21 people18d activeSteady, +2 this week
Posts per day
The posts behind it
...from caches to consumers of their API's [?]). VRAM, meanwhile, still remains the bottleneck for consumers. I think this means those working for Anthropic/OpenAI/etc won't be making as many meaningful advances to cut down VRAM for inferences. Why do thusly if that only allows the...
Feedback wanted: the TypeScript SDK interface for authoring Dags
Airflow Dags are no longer Python only. Language SDKs for **Java**, **Go** and **TypeScript** let you declare a whole Dag (schedule, tasks, options and dependencies) in another language, or implement the stub tasks of a Python Dag in it. The TypeScript SDK ships with Airflow 3.4...
Qwen3.8 Flash Next decodes at 5–9 tok/s on M4 Max (catalog estimates ~60), GPU ~21% busy
...CPU stayed around 0.1-1.5%. The GPU isn't saturated and the CPU isn't the bottleneck either. ## Expected Decode speed in the range of the catalog estimate (~60 tok/s). With 6B active parameters it should be well above the dense 27B on the same machine. ## Notes - Both variants...
Qwen3.8 Flash Next still fails to load on Apple silicon in 0.2.14: "import window: mapped source exceeds admitted import transient"
Follow-up to #227. After updating to 0.2.14, the "prepared N bytes but the ordered program plan charged N-3" error is gone, but both quants of Qwen3.8 Flash Next now fail at a later step. Environment - MacBook Pro M4 Max, 128GB unified memory - Magnitude 0.2.14 (engine artifact...
[Feature]: Publish macOS wheels to PyPI.
...- #35419 - Publish macOS wheels One of the main blockers discussed in #15661 was the lack of macOS CI resources. Both issues were eventually closed automatically due to inactivity. It seems the original infrastructure blocker has been addressed, so it may be worth revisiting...
...and detailed faces onto your mesh, two things Trellis 2's built-in texturing has long struggled with.. # b3d # AI . https:// gachoki.com/ultratex-ai-textur ing-6gb-vram/ .
[Tracking][Help Wanted] TP-agnostic KV offloading
Canonical KV offloading lets compatible engines with different TP sizes share offloaded KV. This tracker covers the remaining work. **Help wanted:** existing PRs need reviews and testing; coverage items below need owners. Comment before starting so we can coordinate....
0.2.8 starts a model too large for free memory and stops it later; 0.2.6 refused it at once
## Environment - Magnitude desktop 0.2.8 (service owner: Desktop), macOS on an Apple M4 Pro with 48 GB - Model: `qwen3.6-35b-a3b:gguf:q5` (listed at 27.5 GB in 0.2.8, 27.9 GB in 0.2.6) - Memory at the time: about 26 GB free and 9 GB inactive (`vm_stat`), swap 5.2 to 6.7 GB in...
...intense prompting. The harnesses are simultaneously too open and restrictive, and annoying. The terms are too overloaded. The models fight with me on highly domain specific knowledge that it's long since compressed or quantized away, until I fill the context with the...
Which Image Gen model is best for infographics?
...keeps constantly changing, running out of limits and other stupid bs like that. Its unreliable. What image gen model is best for the task of complex infographic generation? Also, what model could take the input and actually produce it into a infographics suitable prompt? I...
Tested Mellum2.1-12B-A2.5B on PI Coding Agent - surprisingly usable, but not great at one-shot projects
...a specific folder. It handled the whole thing without any issues.. It does sometimes struggle with unfamiliar workflows, though.. Running it on a laptop with 8GB VRAM and 32GB RAM at 131K context. I'm getting around 40 t/s.. You might ask why I don't use Qwen3.6 35B-A3B instead....
Epyc inference rigs
I posted on here a while back asking for advice on upgrading to an Epyc system. Thanks to all who commented.. I’ve taken the plunge and ordered:. - Epyc 7443 CPU (24 cores, 48 threads, 128 lanes). - H12SSL-NT-B motherboard (the variant with two 10Gb Ethernet ports). - 256GB...
Thank you :) Swift Models hit 2.2 million+ downloads / Early Access to New Models, Free Compute for Researchers
...quantizations, finetunes or just using them. Without all of you guys, we would have had no way to continue our work, and now with the downloads rolling in we are more than happy (and paid hah) to continue training new models and as of recent making other tools for local AI...
`dotProduct` no-rescoring rewrite sorts USearch `1 - dot` DESC: inverted top-k
### Describe what's wrong - A `dotProduct` vector index at default settings returns the least similar shortlist rows, in reverse order, with values reported as `1 - dot`. - Trigger: `vector_search_with_rescoring = 0` and `ORDER BY dotProduct(vec, ref) DESC LIMIT k` on a...
...of vram (which needs a system level flag set to go over 48gb) But the m1 cpu is itself a bottleneck on prefill compared to say an m5, there’s no real getting around it. And the 400mb/s bandwidth starts to hurt without MOE Hoping these model optimisations can see us through to...
...have been labeled things like idk... QZ12345 or HCF1111 . People chose to call 'em "frustration" or "pain" instead. THEY picked those names because they caused the LLM to act in particular ways. You gonna say the actual vectors aren't there in VRAM, just because you don't like...
...those big zooms, vast amounts of data has to be shuttled in and out of the GPU. VRAM is too expensive to keep it all live. There are some other current problems. GPU prices have not come down. If you want a big, high-detailed world, you need a few hundred dollars of GPU. GPUs on...
[Vulkan] Catalog model qwen3.8-27b:gguf:q4 (speculative DFlash2) fails to load: draft convolve kernel uses reserved word 'input'
...**`qwen3.8-27b:gguf:q4`**, whose reviewed configuration uses the **DFlash2** drafter, fails to load. The failure is in the DFlash2 draft kernels: they declare local variables named `input` (in `draft_convolve_input.comp`) and `output` (in the sibling...
PerChannelMinMaxObserver.reset_min_max_vals() drops the dtype passed via factory_kwargs
*Disclosure (per the AI policy): I prepared this report with help from Claude (an AI assistant). I ran the snippet below myself in the environment listed under Versions and pasted its output unmodified.* ### 🐛 Describe the bug `PerChannelMinMaxObserver.reset_min_max_vals()`...
I built my own database GUI for PostgreSQL and MySQL
...to know:. What features do you use most in your database GUI?. What do you find frustrating about existing tools?. What would you add to something like Sqlaris?. Are there any workflows that you think could be much simpler?. I'll be actively improving it, so useful feedback can...
How much space and memory would the next generation ngram models take?
...to SSDs and RAM. This could free up VRAM memory significantly. But, then wouldn't the bottleneck become the RAM and SSD storage? Considering just how much information these models are trained upon, wouldn't even the small 27B models take up hundreds of GB RAM or terabytes of...
Most vector quantizers are the same 6 primitives strung in a different order [R]
...art. So this summer we set out to survey it. We found more papers than we expected, and no way to compare them: every paper measured different metrics, on different datasets, tuned for different hardware. We could not find a single systematic evaluation of the leading methods...
Companies and products named
About this problem
Evidence
22 posts from 21 people in 10 places, about 3 a week over 52 days. Mostly on r/LocalLLaMA, Hacker News comments, magnitudedev/magnitude. Tools named alongside: GitHub, Qwen, vLLM, Pinecone.
Frustration Frustration 2 of 3· Seen on GitHub issues, Reddit, Hacker News, Mastodon
How it was grouped
Posts that state a pain and match the "local-llm" rule. First post Sep 17, 2026, latest Nov 7, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running Ollama locally hits memory and speed limits
47 posts from 45 people on GitHub issues, Reddit.
Running llama.cpp locally hits memory and speed limits
45 posts from 41 people on GitHub issues, Reddit.
Running vLLM locally hits memory and speed limits
21 posts from 20 people on GitHub issues, Reddit.
Running Qwen locally hits memory and speed limits
18 posts from 18 people on GitHub issues, Reddit, Stack Overflow.
Running Hugging Face locally hits memory and speed limits
17 posts from 15 people on GitHub issues, Reddit.
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
Running OpenAI locally hits memory and speed limits
13 posts from 13 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
8 posts from 8 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running PyTorch locally hits memory and speed limits
5 posts from 5 people on GitHub issues, Hacker News.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
History