Running Hugging Face locally hits memory and speed limits
19 posts from 16 people13d activeFading, +0 this week
Posts per day
The posts behind it
FineGrainedFP8 dequantize with dtype=float32 leaves non-quantized weights in bfloat16
### System Info - `transformers` 5.22.0, and `main` at 863d262021 - `torch` 2.13.0+cu130, `accelerate` 1.14.0, `safetensors` 0.8.0 - Python 3.13.13, Linux x86_64, 1x H100 ### Who can help? @SunMarc @MekkCyber ### Reproduction Loading a block-FP8 checkpoint with...
Docs: dead links in quantization guides (fp-quant collection, HQQ blog)
Two dead URLs in `docs/source/en/quantization/`, re-verified with GET: | File | Line | Dead URL | Status | |---|---|---|---| | quantization/fp_quant.md | 46 | https://huggingface.co/collections/ISTA-DASLab/fp-quant-6877c186103a21d3a02568ee | 404 | | quantization/hqq.md | 101 |...
MiniMax-M3-VL: vision attention mixes images across samples in a batch
With `MiniMaxM3SparseForConditionalGeneration` and ordinary SDPA attention, keeping sample A unchanged and replacing only sample B's image changes **A's logits**, including the final text position. A scalar computed only from A's output also has a nonzero gradient with respect...
GLM-5.3-Flash: `f_a_proj`/`f_b_proj` are quantized to FP8 although the checkpoint keeps them in bf16
### System Info - `transformers` version: 5.22.1 (also 5.17.0 and main, 03238b2599) - Platform: Linux-6.8.0-1066-aws-x86_64-with-glibc2.35 - Python version: 3.13.9 - Huggingface_hub version: 1.33.0 - Safetensors version: 0.8.0 - Accelerate version: 1.15.0 - Accelerate config:...
`Mxfp4Config(dequantize=True)` ignores dtype: experts are always dequantized to bfloat16
### System Info - `transformers` main at 763b4ba5378dbbb874aec18fe3b18dccabdd4576 (5.23.0.dev0), also 4.56.2 - `torch` 2.13.0+cu130 (main), 2.9.1+cu128 (4.56.2) - 1x H100, Linux ### Who can help? @SunMarc @MekkCyber ### Information - [ ] The official example scripts - [x] My own...
Qwen3.8 Flash Next fails to load on M5 Max — Metal program_factory workspace mismatch
I encountered this error while trying to load Qwen3.8 Flash Next on my M5 Max (Apple Silicon): internal load failure: failed to qualify program_factory on native (metal) with native invocation workspace: prepared 105 bytes but the ordered program plan charged 102 ## Environment...
spec: DFlash drafters with attention_bias=true (the gpt-oss drafters) fail to load — "wrong number of tensors; expected 123, got 91"
### Summary A DFlash draft model trained with `attention_bias: true` (the gpt-oss drafters, e.g. [z-lab/gpt-oss-120b-DFlash](https://huggingface.co/z-lab/gpt-oss-120b-DFlash)) converts fine with `convert_hf_to_gguf.py`, but `llama-server --spec-type draft-dflash -md .gguf`...
Lossless state-tree weights and KV cache: ~24% smaller BF16 checkpoints, faster inference
### Feature request Support for **state trees** ([paper](https://huggingface.co/papers/2610.31931), [code](https://github.com/ammarali32/StateTree-Representation)), a lossless representation for model weights and the KV cache. Each group of 64 BF16 values shares one exponent...
Qwen4-Exp: eager QSA attention produces NaN logits for left-padded samples
Qwen4-Exp with `attn_implementation="eager"` produces NaN logits at **valid tokens** when a batch contains a left-padded sample. In the same batch, the unpadded sample remains finite. With the same weights and inputs, SDPA is finite; leaving the backend unspecified selects SDPA...
Qwen4-Exp: cache.reset() leaves stale position_ids when reusing DynamicCache or StaticCache
`Qwen4ExpTextModel` stores the full `position_ids` on the `Cache` object, but `cache.reset()` only resets the per-layer state. Reusing that cache for a new request appends the new positions to the previous request's positions. When QSA actually prunes key blocks, this can change...
[Feature Request] Integrated Hugging Face model browser with one-click GGUF download and run
Summary Add a native Hugging Face model browser to the Ollama desktop application, similar to LM Studio's model discovery experience. Users should be able to discover, search, filter, download, and run newly released open-weight models directly from Hugging Face, without waiting...
Request to add d1-3B and d1-omni-600M
They are small decision model with image support. 600M also supports audio. https://huggingface.co/LiquidAI/d1-3B-GGUF https://huggingface.co/LiquidAI/d1-omni-600M-GGUF Currently they don't work with error `{"error":"unsupported decision encoding \"lfm2-d1\""}`. returns
Help with EXL2/3 pls
...get a stop token), often it's just entire word salad.. Now wtf am I doing wrong? How do I run EXL2 or 3 locally?. Screenshot is default bot's response to "Hello".. https://preview.redd.it/k6q7nbylcauh1.jpg?width=1159&format=pjpg&auto=webp&s=feacb06a0a46239f559f3494bbffaa5de2f3fc1...
Metal: draft-mtp allocates up to ~15 GB host (malloc) memory during a ~96k-token session vs ~5 GB without spec (Qwen3.8-27B, 64 GB M4 Max)
...flat. Combined with a fixed ~2.9 GiB GPU-side (shared memory) increase, MTP is basically unusable for long sessions on a 64 GB machine where the no-spec baseline fits comfortably. Setup: llama-server (router mode), master @ fc9ce6b9d. Run A (no spec): llama-server --model...
QuantizedCache exceeds residual_length on multi-token cache updates
### System Info Transformers 5.19.0.dev0 at `6a33cb58b35ddc93a8dadefc908c25637ed18da0`, Python 3.12.3, PyTorch 2.14.0+cpu, optimum-quanto 0.2.7, Linux x86_64 under WSL2. No checkpoint downloads or source modifications. ### Who can help? _No response_ ### Information - [ ] The...
Add support for LFM2/2.5 in GGUF loading path
### Feature request Add LFM2/2.5 dense models (`lfm2` architecture) in the new GGUF loading path (`GGUF_ARCHS` / `GGUF_CONFIG_ARCHS`). Today only `qwen35` and `qwen35moe` use it, so LFM2 GGUF files go through the old loader. ### Motivation Follow-up to #49337, where @SunMarc...
Vulkan ErrorDeviceLost (vk::Queue::submit) on AMD Radeon AI PRO R9700 when SAM/ReBAR enabled
### Name and Version version: 0.5.0-dev (build 11212, commit da6c28eb1) built with Clang 20.1.8 for Windows x86_64 ### Operating systems Windows ### GGML backends Vulkan ### Hardware CPU: AMD Ryzen 7 5800X GPU affected: AMD Radeon AI PRO R9700 (32 GB / 32624 MiB), selected as...
[Model Proposal] DualBigLittle-MoE: Tri-Tier Hierarchical MoE with Dual VRAM Cores & Streaming Micro-Experts
### Model description Hi Transformers Team @ArthurZucker @LysandreJik, I have developed and verified a novel **Tri-Tier Hierarchical MoE** architecture called **DualBigLittle-MoE**, designed to break the consumer VRAM capacity wall while eliminating multitask negative transfer...
Validate scale file paths in quantized HuggingFace checkpoint indexes
> **AI-assisted issue conversion (Codex).** Original report and reproduction by @SounLabs; all credit belongs to the original reporter. > > Source: https://github.com/pytorch/pytorch/security/advisories/GHSA-f569-37c9-8pv7 > > The reporter identifies an unchecked `weight_map`...
Companies and products named
About this problem
Evidence
19 posts from 16 people in 6 places, about 3 a week over 43 days. Mostly on huggingface/transformers, ggml-org/llama.cpp, ollama/ollama. Tools named alongside: GitHub, PyTorch, NVIDIA, llama.cpp.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit
How it was grouped
Posts that state a pain and match the "local-llm" rule about Hugging Face. First post Sep 29, 2026, latest Nov 10, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running llama.cpp locally hits memory and speed limits
54 posts from 50 people on GitHub issues, Reddit, Stack Overflow.
Running Ollama locally hits memory and speed limits
48 posts from 46 people on GitHub issues, Reddit.
Running vLLM locally hits memory and speed limits
27 posts from 26 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
24 posts from 23 people on GitHub issues, Reddit, Hacker News.
Running Qwen locally hits memory and speed limits
20 posts from 20 people on GitHub issues, Reddit, Stack Overflow.
Running NVIDIA locally hits memory and speed limits
17 posts from 17 people on GitHub issues.
Running OpenAI locally hits memory and speed limits
14 posts from 14 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
11 posts from 11 people on GitHub issues, Hacker News.
Running PyTorch locally hits memory and speed limits
10 posts from 6 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other Hugging Face problems
Hugging Face users keep filing bugs
Top post: "get_uniform_frame_indices can return too many or out-of-range frame indices", one of 45 posts from 39 people on GitHub issues, Community forums, Stack Overflow.
Hugging Face upgrades break working setups
22 posts from 19 people on GitHub issues, Community forums, Reddit.
Fine-tuning Hugging Face is costly and unpredictable
12 posts from 12 people on GitHub issues, Reddit, Community forums.
Trouble with Hugging Face fail
Top post: "All paid GPU pools fail to start: MOUNT_ATTR_IDMAP error at container init (Exit code 128)", one of 8 posts from 5 people on GitHub issues, Community forums.
Trouble with Hugging Face generate
Top post: "prefill_chunk_size drops image inputs, VLMs generate without the image", one of 6 posts from 5 people on GitHub issues, Community forums.
Trouble with Hugging Face docs
Top post: "Docs: broken download/example asset URLs in code examples (7 URLs return 404)", one of 5 posts from 4 people on GitHub issues.
Trouble with Hugging Face token
Top post: "<|image|> token has special=False in Rust tokenizer backend, should it be True?", one of 4 posts from 4 people on GitHub issues.
Trouble with Hugging Face logits
Top post: "`DogeCDMoE` router `.view(2, bsz * seq_len, -1)` mixes tokens: with `is_moe=True`, logits depend on future", one of 4 posts from 4 people on GitHub issues.
Trouble with Hugging Face gemma
Top post: "gemma 4 can not load audio file and produce Audio features and audio tokens do not match, tokens: 31,", one of 4 posts from 4 people on GitHub issues.
Trouble with Hugging Face python
Top post: "import transformers spends 1-3 s in an eager importlib.metadata.packages_distributions() when torch >= 2.14", one of 4 posts from 4 people on GitHub issues, Community forums.
Trouble with Hugging Face space
Top post: "Unable to start any space", one of 3 posts from 3 people on Community forums.
Hugging Face runs out of context on real work
3 posts from 3 people on GitHub issues.
History