Problemsby GlyphNVIDIA users keep filing bugs
41 posts from 38 people22d activeRising, +6 this week
Posts per day
The posts behind it
Proposal: FastMTP speculative draft graph and scatter
**Status & Context:** Experimental research; not ready for upstream submission. A scoped search did not identify an equivalent upstream issue or PR, but standalone benefit has not yet been demonstrated. **Motivation:** Multi-token prediction (MTP) drafting and verification...
Make a generic accelerator stream sanitizer
Following the same design as torch/cuda/_sanitizer.py . Make a version that relies on the generic Stream/Event and allocator to provide the same behavior. We should also have simple hooks such that backend specific code can trigger behavior properly. We most likely want it to...
Problems in argument-passing with quoting in pwsh
### Description Python extension will add uneeded quote when using pwsh, which cause problems This is the command invoked: & 'C:\Program Files\Python314\python.exe' '"h:/Programing/Python/VSCode Quoting Problems/This is a Test.py"' ## Steps to reproduce: 1. Create a file called...
WSL2: GPU memory on NVIDIA RTX Spark N1X laptops is capped at the guest's free RAM
### What is the issue? On an RTX Spark N1X laptop (HP OmniBook X 14, 128 GB unified memory, 80 GB dedicated), Ollama in WSL2 offloads only part of a model that fits in GPU memory. Firmware dedicates 80 GB to the GPU and Windows sees the remaining 46.4 GiB. I chose 80 GB rather...
[Bug]: nvidia_riva sends no authorization header on plaintext gRPC and duplicates it over TLS (riva-client version dependent)
...entirely.. Related gaps (optional, same area) No CA option: with use_ssl=True , there is no way to pass a CA certificate. ssl_root_cert / ssl_cert is never forwarded to Auth , so private-CA endpoints only work by setting GRPC_DEFAULT_SSL_ROOTS_FILE_PATH on the proxy. A...
Can't "Keep" or "Undo" chat file changes
### Description If a forked chat changes a file, the pending file changes bar gets stuck and neither "Keep" nor "Undo" work. The error message is: > Could not keep changes to BugRepo.txt: Changes to 'BugRepo.txt' can't be accepted because another chat in this session also...
[hardware support] Moore Threads GPU (MUSA) support in flash linear attention
## this is a pre-PR issue ## Feature request ### Background `is_flash_linear_attention_available()` currently allows CUDA, XPU, and MLU backends, but it does not consider MUSA (Moore Threads) environments. The repository already provides `is_torch_musa_available()` in...
D3D12 pipeline-library static-sampler issue with NVIDIA RTX 5060, 621.01 / 32.0.16.2101
[RFC]: Attention Backend Test Suites
...PR should prove. Trying the standard on one backend showed the gap. ROCM_AITER_FA had no failing tests, yet a suite built to the standard found seven backend defects that tests did not catch: - wrong results under model runner v2; - logits soft cap, sinks and ALiBi dropped in...
RE::-Overheating and failing the Nvidia Fieldtest Diag
[FEATURE] Native NVIDIA NIM (nvidia_nim) provider via OpenAICompatibleCompletion
### Feature Area Integration with external tools ### Is your feature request related to a an existing bug? Please link it here. NA ### Describe the solution you'd like crewAI already routes DeepInfra, Cerebras, DeepSeek, OpenRouter and others natively through...
CUDA max_unpool2d: illegal memory access when the input has more than INT_MAX elements
### 🐛 Describe the bug `F.max_unpool2d` on CUDA hits an illegal memory access when the input has more than 2³¹ elements, even when every index is valid. The driver logs an Xid 31 MMU fault (`FAULT_PDE ACCESS_TYPE_VIRT_READ`). Reproduced on an H100 80GB with 2.14.1+cu130 and...
`HuggingFacePipeline.from_model_id` crashes on device strings and misplaces models on non-CUDA accelerators (e.g. Intel XPU)
### Submission checklist - [x] This is a bug, not a usage question. - [x] I added a clear and descriptive title that summarizes this issue. - [x] I used the GitHub search to find a similar question and didn't find it. - [x] I am sure that this is a bug in LangChain rather than...
Qwen3-VL patch embedding: Conv3d 58.5s vs equivalent matmul 0.2ms on H800
## System Info NVIDIA H800 (SM90), torch 2.9.1+cu128, cuDNN 9.10.2 (91002), transformers 5.20.1, bf16. ## Observed behavior Qwen3VLVisionPatchEmbed applies Conv3d to (8192,3,2,16,16), kernel=stride=(2,16,16), out_channels=1152. In my environment, synthetic tensors with this...
CI: Idefics3 test_model_parallel_beam_search fails with CUDA device-side assert
...exact `inputs_dict` from `prepare_config_and_inputs_for_generate` is not provided; the failing call was not run here. The CI failure occurs at: after `new_model = model_class.from_pretrained(tmp_dir, device_map="auto")`. Failing CI test:...
CI: Idefics3 test_onnx_export_static fails with torch.AcceleratorError CUDA device-side assert
...CUDA device-side assert during tensor creation in test setup, and no snippet was run. Failing CI test: `tests/models/idefics3/test_modeling_idefics3.py::Idefics3ForConditionalGenerationModelTest::test_onnx_export_static` [Failure...
CI: Idefics3 test_model_parallelism fails with torch.AcceleratorError CUDA device-side assert
...before any checkpoint is loaded and the exact `model_tester` inputs are not provided. Failing CI test: `tests/models/idefics3/test_modeling_idefics3.py::Idefics3ForConditionalGenerationModelTest::test_model_parallelism` [Failure...
CUDA SymmetricMemory: rank-local rendezvous failure leaves peer resources and stalls other ranks
### Describe the bug I tested the behavior when a CUDA SymmetricMemory rendezvous fails on a specific rank. By artificially injecting a peer-import or peer-map failure on Rank 0, Rank 1 returned after approximately 301 seconds, with the imported resources remaining unreleased....
benchmark_utils synchronizes and profiles only CUDA
### 🚀 The feature, motivation and pitch `torch._functorch.benchmark_utils` treats `devices=["cuda"]` as the sync path and always records `ProfilerActivity.CUDA`. On a renamed PrivateUse1 backend this follows the CUDA sentinel instead of the current accelerator, and the profiler...
[RFC] Make generic code paths device-agnostic: remove hardcoded CUDA-specific assumptions to support third-party accelerator backends
...and silently falls back to CPU state. **This is a latent correctness bug, not just a missing feature**; - Closed, literal capability sets are **imported by value** into multiple namespaces, so OOT backends must monkey-patch multiple places to opt in - fragile and error-prone....
Unable to login in nvidia career page
Kubelet image-pull errors do not distinguish NotFound from an anonymous fallback failure
...behavior, original errors, and ErrImagePull/ImagePullBackOff semantics. ### How can we reproduce it (as minimally and precisely as possible)? 1. Use an AKS cluster with the Azure exec credential provider and give its kubelet managed identity `AcrPull` on an ACR registry with...
Any Advice for what Industry is looking for in ML PhDs? [D]
I'm a first year ML PhD at a T5 ML program in the US. I managed to get a first author paper accepted to NeurIPS Main Track last month and thought I'd start applying for PhD research internships since I saw a lot of them required a "top" venue publication.. But I've just been...
[distributed] _check_rng_sync raises NotImplementedError for non-philox generator devices (NPU/MTIA/HPU/...): should fall back to full-state check
...... - hits the `else` branch and raises `NotImplementedError`, so `_check_rng_sync` is unusable on those backends. The `elif "cpu"` branch is not fundamentally CPU-specific: it simply gathers each rank's full generator state tensor and compares `torch.hash_tensor` values. That...
Are there instructions for building from source?
I'm on Bluefin Linux (x86_64 / NVIDIA), which doesn't make installing pre-built `.rpm` or `.deb` packages easy. is there a set of instructions for building this from source?
Consistent Lazy API engine instantiation for Python unit tests
...hard-code the `engine` parameter to `collect*`, `sink_*`, etc. calls which makes it difficult to "inject" a different engine for testing. It appears that the `polars.Config.set_engine_affinity` might be a nice way to generically ensure that these collect et. al. calls run with a...
CUDA: Pascal DP4A MMQ config uses I=64 for Q8_0; I=128 gives +12% pp512 on sm_61
...| Q6_K | **−51.7%** | −0.1% | `I=160` regresses 45.7% even on Q8_0, and `I>=192` fails to compile - Pascal shared-memory limit. 128 is the largest value that fits, and it is what `mmq-config-ampere.cuh` already uses for this type. ### Still applies on current master Re-checked...
Reduce hashing overhead for CUDA and XPU streams
Opening this issue following the discussion in [PR \#189260 (https://github.com/pytorch/pytorch/pull/189260)]( ). `torch.cuda.Stream.__hash__` currently computes `hash((self.cuda_stream, self.device))`, and the XPU equivalent uses `hash((self.sycl_queue, self.device))`. Each...
Nvidia’s erroneous paper accepted as ICML’s spotlight [D]
...their released everything in pre training phase, post training phase and evaluation is buggy and is wrong. . The code doesn’t seem AI written cause its (very) bad written. And now after seeing all those bugs the results in paper makes sense.. My question is:. WTF!. How it did...
blackwell gpus have PCIe5 issues?
...but no success. My code is fine.. Then I investigate.... Apparently this is just problem with Blackwell we just accept?. https://en.gamegpu.com/news/zhelezo/rtx-5070-rtx-5080-i-rtx-5090-prodolzhayut-zavisat-iz-za-neizvestnogo-sboya-pcie-gen-5 . I searched this sub and reddit,...
[Feature] Decouple log_sigmoid_forward buffer caching from hard-coded CUDA/XPU for privateuse1 backends
## Motivation `log_sigmoid_forward` uses `if self.is_cuda or self.is_xpu:` to decide whether to cache the intermediate buffer (`z = exp(-|x|)`) in the forward output. CUDA/XPU skip caching to save device memory (backward recomputes). On a `privateuse1` backend (Ascend NPU),...
`clef-flash` decision model always fails on `/v1/systemone` — "Clef: non-finite logit" (CUDA) / "Clef: cannot open model" (CPU)
The decision model `clef-flash` (Q8_0, 9.1B) always fails on the `/v1/systemone` endpoint. The same model works fine on `/v1/chat/completions`, and the larger `clef:27b` (Q4_K_M) works on `/v1/systemone` on the same machine. The error occurs on the very first forward pass...
Go back to the old agent flow
Type: Feature Request I don't like the new agent workflow, where it just changes my code and doesn't show me or ask me to approve it. Go back to proposed edits in green with Keep/Undo approval. Or at least have a way to choose which one we want to use. VS Code version: Code...
What should i do?
I’ve always been interested in tech and how computer science seems. But an issue is i’m terrible at math. I always see these people doing internships/working with these massive tech companies like nvidia, roblox, roku, google, and so many more great tech companies. I’ve dreamed...
[CD] Remove temporary cuBLAS override for CUDA 13.4 Linux wheels
We are holding the full CUDA 13.4 Update 1 move because its cuSOLVER 12.3.4.7 can cause **link errors when previously built libraries that require versioned cuSOLVER symbols are relinked against it**. This is a specific link-time compatibility issue; it does not mean every build...
[numeric] fp32 CPU vs fp32 CUDA (TF32 off): single-op matches but final grad-norm and logits-norm diverge after 20 deterministic steps (ResNet34, AdamW lr=0.1, clip 5.0)
### 🐛 Describe the bug I compared a **fp32 CPU** training run against a **fp32 CUDA** training run of the same ResNet34 (torchvision, CIFAR stem: conv1 3x3 stride 1, maxpool replaced by Identity) on identical synthetic deterministic data, single node, with TF32 **disabled**...
[numeric] fp32 CPU vs fp32 CUDA (TF32 off): single-op matches to 1e-6 but final loss/grad-norm/logits diverge after 20 deterministic steps (ResNet18, SGD lr=1.0)
### 🐛 Describe the bug I compared a **fp32 CPU** training run against a **fp32 CUDA** training run of the same ResNet18 (torchvision, CIFAR stem: conv1 3x3 stride 1, maxpool replaced by Identity) on identical synthetic deterministic data, single node, with TF32 **disabled**...
Intel Arc (B70, level_zero): Conv1d/Conv2d/matmul return NaN in bf16 and fp16; fp32 and CUDA clean
...clean on CUDA, and fp32 is clean on the same XPU device. This makes reduced precision unusable on Arc for any model whose forward pass contains a convolution or an attention matmul. `nn.Linear`, `softmax`, `gelu` and `layernorm` are all clean on the same device and in the same...
How to use onnxruntime on same GPU as graphics api i.e opengl
...order to prepare frame for rendering. Which leads to application having low frame rate.. Is there a way to tell if a GPU will handle context switching between ML and graphics or is the only option to optimise the underlying model ops to run faster, I'm specifically referring to...
[1.36] Kubelet retries DRA health streams for plugins that do not advertise health support
...retrying until the plugin re-registers. This is the behavior in Kubernetes 1.37. ### How can we reproduce it (as minimally and precisely as possible)? This was reproduced with a kind v1.36.1 cluster using the head of:...
Bug about transformers==5.16.1 while loading 'nvidia/audio-flamingo-next-hf' model.
### System Info ### Who can help? _No response_ ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ###...
Companies and products named
About this problem
Evidence
41 posts from 38 people in 17 places, about 4 a week over 71 days. Mostly on pytorch/pytorch, huggingface/transformers, microsoft/vscode. Tools named alongside: PyTorch, GitHub, Hugging Face, Ollama.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit, Community forums, Stack Overflow
Summary, model-written
NVIDIA users encounter endpoint model failures, llama-server startup assertion errors, unwanted automatic code changes, and cuSOLVER link errors.
How it was grouped
Posts about NVIDIA that state a pain but match no known issue yet. First post Aug 28, 2026, latest Nov 6, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
Other NVIDIA problems
Running NVIDIA locally hits memory and speed limits
16 posts from 16 people on GitHub issues.
Trouble with NVIDIA windows
Top post: "Windows CUDA prebuilt binaries stuck at b11828 while release tags keep advancing (b11846..b11852)", one of 3 posts from 3 people on GitHub issues.
NVIDIA GPU access and drivers are painful
12 posts from 12 people on GitHub issues, Mastodon, Reddit.
NVIDIA vulnerabilities are hard to triage and patch
77 posts from 77 people on Glyph Ibis.
NVIDIA upgrades break working setups
11 posts from 11 people on GitHub issues.
Trouble with NVIDIA account
Top post: "NVIDIA Careers Workday Account Locked - Unable to Complete Employee Referral Application", one of 9 posts from 8 people on Community forums.
NVIDIA lock-in makes switching costly
5 posts from 5 people on GitHub issues, Hacker News, Bluesky.
NVIDIA runs out of memory
5 posts from 5 people on GitHub issues.
Trouble with NVIDIA slower
Top post: "CUDA training loop runs ~50x slower while a DataLoader iterates in a background thread, since 2.10.0", one of 4 posts from 4 people on GitHub issues.
Trouble with NVIDIA process
Top post: "`HuggingFaceEmbeddings(multi_process=True)` starts CPU workers on non-CUDA accelerators", one of 3 posts from 3 people on GitHub issues.
Trouble with NVIDIA phone
Top post: "Unable to receive OTP for Vietnam (+84) phone verification on build.nvidia.com", one of 3 posts from 3 people on Community forums.
Fine-tuning NVIDIA is costly and unpredictable
3 posts from 3 people on GitHub issues.
History