Problemsby GlyphNVIDIA upgrades break working setups
11 posts from 11 people10d activeFading, +0 this week
Posts per day
The posts behind it
Expose opt-in draft acceptance statistics for assisted decoding
...in one verification step. But if the system isn't getting the expected speedup, it's difficult to know exactly what's going wrong just by looking at generation latency. This was something I ran into while benchmarking SmolLM2-360M-Instruct as the target model and...
Use device-side assertions (CUDA_KERNEL_ASSERT2) in user-reachable index kernels
...already exist and are compiled into every CUDA and ROCm build. They avoid the trap: the failing thread records the failure in a managed buffer and returns, and the host raises at the next CUDA check. Today only the c10/cuda test kernels use them. I'd like to use them in the...
Python kernels silently drop explicitly-passed float args that are IEEE-equal to the schema default (-0.0 to +0.0)
## Description When a custom operator is registered with a Python implementation (`torch.library.impl` / `PythonKernelHolder`) and its schema declares a **float default value**, any argument the caller passes that is **IEEE-equal to the default but bit-different** is silently...
[PodLevelResources] Pods without pod-level resources get an extra status PATCH because status.resources changes after Running
### What happened? With `InPlacePodLevelResourcesVerticalScaling` enabled (beta, on by default since 1.36), the kubelet changes pod-level `status.resources` after a pod starts, even though nothing was resized. If the pod lives past the next kubelet sync, this is a separate...
v0.40.5 on Jetson Orin Nano 8GB (JetPack 7 / CUDA 13): gemma4:e4b-it-qat fails at num_ctx 16384 (NvMap error 12), 2 tok/s at 8192; fine on v0.35.1
...GPU at 8192; I suspect partial CPU offload but have not confirmed it. ## Log from the failing 16384 start (v0.40.5) Runner command line (v0.40.5, model paths removed): Note: v0.35.1 also starts its runner with `--load-mode dio` here (every runner start in my journal), so direct...
bug(cuda): A100 PCIe bus loss during Qwen35 warm-up/inference on 0.34.4 and 0.35.0
...configuration. **This is a suspected regression, not an isolated causal change.** Failing and passing tests were in different host boots. An intermittent card/slot/power fault, driver/IOMMU state or release/build interaction remains possible. We are reporting the observed...
[BUG][Spark] Delta Spark 3.3.3 deletion-vector bytecode is incompatible with Java 8
## Bug #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ### Describe the problem The published Delta Spark 3.3.3 artifacts contain Java 8 class files, but the deletion-vector deserializer...
[inductor][cutlass] CUTLASS cache key ignores the XPU toolkit, and the directory is chosen by a build-time flag
### 🐛 Describe the bug Two related defects make the CUTLASS backend use - or fail to use - the wrong CUTLASS checkout. Both are silent. **1. The cache key hashes the CUDA checkout even when XPU is used.** `cutlass_key()` always hashed `config.cutlass.cutlass_dir`, but the XPU...
Text Editor tab bar and inactive tab color has been messed up by latest update
Type: Bug Using the default Modern Dark theme, the color of the inactive tabs as well as the the tab bar now has this ugly, washed out bright gray color reducing readability of the inactive tabs. This is a regression from the pre-update theme where the inactive tab and tab bar...
Cache.crop does not roll back linear-attention recurrent states — block/speculative verification on GDN hybrids (Qwen3-Next / Qwen3.5) is not lossless with DynamicCache
## System info - transformers 5.16.1 (also read 5.15.1 source), torch 2.11, CUDA (GB10 / DGX Spark); model type `qwen3_5` (hybrid: 48 Gated-DeltaNet layers `LinearAttentionLayer` + 16 attention layers `DynamicLayer` in `DynamicCache(config=...)`) ## Description...
[RFC] Mamba2: exact-replay decode so that prefill, chunked prefill and decode produce identical bits
## Design update (2026-09-06) Following maintainer feedback on #55525 (batch invariance is organised per kernel, not per model), the implementation is redone as a single replacement PR and the following decisions of this RFC are superseded. A separate PR pinning the SSD kernels'...
Companies and products named
About this problem
Evidence
11 posts from 11 people in 7 places, about 1 a week over 53 days. Mostly on pytorch/pytorch, huggingface/transformers, ollama/ollama. Tools named alongside: PyTorch, Hugging Face, GitHub, Ollama.
Frustration Frustration 0 of 3· Seen on GitHub issues
How it was grouped
Posts that state a pain and match the "breaking-changes" rule about NVIDIA. First post Sep 6, 2026, latest Oct 28, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
GitHub upgrades break working setups
87 posts from 75 people on GitHub issues, Hacker News, Stack Overflow.
Upgrades and new releases break working setups
76 posts from 62 people on GitHub issues, Reddit, App Store reviews.
Claude Code upgrades break working setups
67 posts from 60 people on GitHub issues, Hacker News, Reddit.
Codex upgrades break working setups
65 posts from 58 people on GitHub issues, Reddit.
DuckDB upgrades break working setups
37 posts from 28 people on GitHub issues.
Claude upgrades break working setups
35 posts from 34 people on GitHub issues, Reddit, Hacker News.
ClickHouse upgrades break working setups
31 posts from 20 people on GitHub issues.
Anthropic upgrades break working setups
27 posts from 22 people on GitHub issues.
PyTorch upgrades break working setups
26 posts from 21 people on GitHub issues, Reddit.
ChatGPT upgrades break working setups
26 posts from 26 people on GitHub issues, Community forums, App Store reviews.
OpenAI upgrades break working setups
25 posts from 25 people on GitHub issues, Reddit, Community forums.
Open WebUI upgrades break working setups
25 posts from 2 people on GitHub issues.
Other NVIDIA problems
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
NVIDIA users keep filing bugs
Top post: "`clef-flash` decision model always fails on `/v1/systemone`, Clef: non-finite logit (CUDA) / Clef: cannot", one of 41 posts from 38 people on GitHub issues, Reddit, Community forums.
Trouble with NVIDIA windows
Top post: "Windows CUDA prebuilt binaries stuck at b11828 while release tags keep advancing (b11846..b11852)", one of 3 posts from 3 people on GitHub issues.
NVIDIA GPU access and drivers are painful
12 posts from 12 people on GitHub issues, Mastodon, Reddit.
NVIDIA vulnerabilities are hard to triage and patch
77 posts from 77 people on Glyph Ibis.
Trouble with NVIDIA account
Top post: "NVIDIA Careers Workday Account Locked - Unable to Complete Employee Referral Application", one of 9 posts from 8 people on Community forums.
NVIDIA lock-in makes switching costly
5 posts from 5 people on GitHub issues, Hacker News, Bluesky.
NVIDIA runs out of memory
5 posts from 5 people on GitHub issues.
Trouble with NVIDIA slower
Top post: "CUDA training loop runs ~50x slower while a DataLoader iterates in a background thread, since 2.10.0", one of 4 posts from 4 people on GitHub issues.
Trouble with NVIDIA process
Top post: "`HuggingFaceEmbeddings(multi_process=True)` starts CPU workers on non-CUDA accelerators", one of 3 posts from 3 people on GitHub issues.
Trouble with NVIDIA phone
Top post: "Unable to receive OTP for Vietnam (+84) phone verification on build.nvidia.com", one of 3 posts from 3 people on Community forums.
Fine-tuning NVIDIA is costly and unpredictable
3 posts from 3 people on GitHub issues.
History