Running PyTorch locally hits memory and speed limits
10 posts from 6 people6d activeRising, +5 this week
Posts per day
The posts behind it
[quantized] Empty BatchNorm inputs ignore requested output quantization parameters
### 🐛 Describe the bug ### Description `quantized::batch_norm1d`, `batch_norm2d`, and `batch_norm3d` return `qx.clone()` when the input is empty. This preserves the input quantization parameters instead of using the requested `output_scale` and `output_zero_point`. ###...
[quantized] Embedding.from_float fails to preserve quint4x2 dtype
### 🐛 Describe the bug ### Description `nn.quantized.Embedding.from_float` fails when quantizing an Embedding with `torch.quint4x2` weights. The resulting weight is correctly quantized to `quint4x2`, but the created Embedding module retains the default `quint8` dtype. Its...
[quantized] PReLU incorrectly scales positive inputs when weight scale is small
### 🐛 Describe the bug ### Description Quantized PReLU incorrectly scales positive inputs when the weight quantization range cannot represent 1.0. The positive branch of PReLU should be an identity operation, but the CPU kernel quantizes 1.0 using the weight's scale and zero...
[quantized] EmbeddingBag silently ignores the mode argument
### 🐛 Describe the bug ### Description `nn.quantized.EmbeddingBag` silently ignores its `mode` argument and always performs sum reduction. The `mode` is stored by the constructor and copied by `from_float`, but `forward()` hardcodes `0` when calling both `embedding_bag_byte`...
[quantization] Quantized MultiheadAttention uses bias_k qparams for bias_v
### 🐛 Describe the bug ### Description `nn.quantized.MultiheadAttention.from_observed()` incorrectly uses `bias_k` to calculate the quantization parameters for `bias_v`. When the two biases have significantly different value ranges, `bias_v` is severely quantized using...
torch.equal still overreads quint2x4 storage in 2.11.0 (follow-up to #162801)
### Bug description `torch.equal()` appears to read past the backing storage of CPU `quint2x4` tensors. I can reproduce incorrect comparisons with the PyTorch 2.11.0 macOS arm64 CPU wheel. A separate guard-page test crashes when the read crosses the storage boundary. This looks...
[FSDP2] uneven-sharding check in `all_gather_inputs` rejects multi-tensor `fsdp_pre_all_gather` payloads (quantized weights)
### 🐛 Describe the bug Under uneven dim-0 sharding, `FSDPParam.all_gather_inputs` (`torch/distributed/fsdp/_fully_shard/_fsdp_param.py`) rejects **any multi-tensor payload** returned by `fsdp_pre_all_gather`, even when every component is correctly padded along the sharded dim:...
...fine-tuning/evaluation workflows. Best fit: startups or teams with ML/AI models that are too slow, too expensive to serve, hard to deploy, or losing accuracy after quantization/compression. Interested in freelance, consulting, part-time, or full-time opportunities around model...
[torch.compile] AssertionError "n=copy_, n.args[0]=dequantize" when a graph input is mutated in place after torch.dequantize on a non-quantized tensor
### 🐛 Describe the bug Mutating a graph input in place after calling `torch.dequantize` on it (with a regular, non-quantized float tensor) works in eager mode but fails under `torch.compile` with an AOTAutograd functionalization assertion. The assertion comes from...
[Bug]: CPU MoE kernel produces NaN logits under torch.compile for Qwen3_5MoeForConditionalGeneration — reproduces with AND without GPTQ-Int4 quantization, fixed by --enforce-eager
...mode worth flagging on its own - a healthy pod/process gives zero signal that generation is broken. **Workaround: `--enforce-eager` completely fixes it, on both checkpoints.** With the identical checkpoint, config, and request, eager mode produces correct, coherent generations...
Companies and products named
About this problem
Evidence
10 posts from 6 people in 3 places, about 1 a week over 71 days. Mostly on pytorch/pytorch, vllm-project/vllm, Hacker News comments. Tools named alongside: OpenAI, vLLM, Hugging Face.
Frustration Frustration 1 of 3· Seen on GitHub issues, Hacker News
Summary, model-written
PyTorch users encounter bugs including graph input compilation assertions, FSDP payload rejections, and out-of-bounds memory reads during tensor comparisons.
How it was grouped
Posts that state a pain and match the "local-llm" rule about PyTorch. First post Sep 6, 2026, latest Nov 15, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Running llama.cpp locally hits memory and speed limits
54 posts from 50 people on GitHub issues, Reddit, Stack Overflow.
Running Ollama locally hits memory and speed limits
48 posts from 46 people on GitHub issues, Reddit.
Running vLLM locally hits memory and speed limits
27 posts from 26 people on GitHub issues, Reddit.
Running LLMs locally hits memory and speed limits
24 posts from 23 people on GitHub issues, Reddit, Hacker News.
Running Qwen locally hits memory and speed limits
20 posts from 20 people on GitHub issues, Reddit, Stack Overflow.
Running Hugging Face locally hits memory and speed limits
19 posts from 16 people on GitHub issues, Reddit.
Running NVIDIA locally hits memory and speed limits
17 posts from 17 people on GitHub issues.
Running OpenAI locally hits memory and speed limits
14 posts from 14 people on GitHub issues.
Running DeepSeek locally hits memory and speed limits
11 posts from 11 people on GitHub issues, Hacker News.
Running Claude Code locally hits memory and speed limits
6 posts from 6 people on GitHub issues.
Running GitHub locally hits memory and speed limits
4 posts from 4 people on Reddit, GitHub issues.
Running Perplexity locally hits memory and speed limits
3 posts from 3 people on GitHub issues.
Other PyTorch problems
PyTorch writes buggy code that is hard to review
2 posts from 1 person on GitHub issues. Unverified.
PyTorch users keep filing bugs
Top post: "Meta-issue: torch.compile gaps found via the CPython test suite", one of 50 posts from 41 people on GitHub issues, Reddit, Stack Overflow.
PyTorch upgrades break working setups
29 posts from 23 people on GitHub issues, Reddit.
PyTorch GPU access and drivers are painful
17 posts from 17 people on GitHub issues, Reddit.
Trouble with PyTorch inductor
Top post: "`[Inductor][CPU] torch.var returns NaN for large-magnitude constant float64 inputs while eager returns 0`", one of 14 posts from 11 people on GitHub issues.
PyTorch builds are slow or flaky
11 posts from 2 people on GitHub issues.
Trouble with PyTorch true
Top post: "PaddleOCR-VL: position embedding interpolation changed from align_corners=False to True", one of 10 posts from 7 people on GitHub issues.
PyTorch deprecations force unplanned migrations
8 posts from 5 people on GitHub issues.
Trouble with PyTorch input
Top post: "Generation crashes with encoder_repetition_penalty when encoder_outputs are provided without input_ids", one of 7 posts from 7 people on GitHub issues.
Fine-tuning PyTorch is costly and unpredictable
5 posts from 5 people on GitHub issues, Reddit.
Trouble with PyTorch dynamic
Top post: "DISABLED test_weight_norm_1d_and_reduced_dtypes_dynamic_shapes (__main__.DynamicShapes)", one of 5 posts from 3 people on GitHub issues, Reddit.
Trouble with PyTorch windows
Top post: "Bundled cuDNN fails on first use since #202206 when no cuDNN is on PATH (Cannot load symbol cudnnGetVersion)", one of 4 posts from 4 people on GitHub issues.
History