Problemsby IbisFine-tuning PyTorch is costly and unpredictable
MLOps2 posts from 2 people2d active+00 posts in the last 7 days, 1 the 7 days before (fading)Unverified
Posts per day
The posts behind it
2, newest first[SDPA] cuDNN backward returns NaN dQ for head_dim=256 with a bool attn_mask and short sequences on Blackwell (B300, sm_103), torch 2.14.1+cu130 / cuDNN 9.24...32, 64, 96, 102}. L = 63, 65 and everything from 127 to 2048 are fine. Repro: Output: Workaround: `torch.backends.cuda.enable_cudnn_sdp(False)`. With it, training ran 150+ steps without NaN at the same speed. Related: - #199731 (closed): NaN dQ with an additive mask on H200,...pytorch/pytorch · Ingvarstep1 repliesOpen ↗Issue doing PEFT on Embedding Gemma w/ new classifier head### System Info Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points. - `transformers` version: 5.16.1 - Platform: Linux-6.6.122+-x86_64-with-glibc2.39 - Python version: 3.13.15 - Huggingface_hub version: 1.29.0 - Safetensors version: 0.8.0 -...huggingface/transformers · sr54342 repliesOpen ↗
Companies and products named
About this problem
Evidence
2 posts from 2 people in 2 places, about 1 a week over 20 days. Mostly on huggingface/transformers, pytorch/pytorch. Tools named alongside: NVIDIA, GitHub, Tesla, Hugging Face.
Frustration Frustration 0 of 3· Seen on GitHub issues
How it was grouped
Posts that state a pain and match the "fine-tuning" rule about PyTorch. First post Sep 20, 2026, latest Oct 9, 2026. Unverified: fewer than 3 posts, or one person or place only. Method
The same problem elsewhere
Fine-tuning Hugging Face is costly and unpredictableFine-tuning is costly and results are unpredictableFine-tuning Qwen is costly and unpredictableFine-tuning llama.cpp is costly and unpredictableFine-tuning OpenAI is costly and unpredictableFine-tuning vLLM is costly and unpredictableFine-tuning GitHub is costly and unpredictableFine-tuning Claude Code is costly and unpredictableFine-tuning Gemini is costly and unpredictableFine-tuning Claude is costly and unpredictableFine-tuning NVIDIA is costly and unpredictableFine-tuning ChatGPT is costly and unpredictable
Other PyTorch problems
PyTorch users keep filing bugsTrouble with PyTorch inputTrouble with PyTorch truePyTorch deprecations force unplanned migrationsPyTorch upgrades break working setupsTrouble with PyTorch workerPyTorch builds are slow or flakyTrouble with PyTorch inductorPyTorch GPU access and drivers are painfulTrouble with PyTorch testPyTorch vulnerabilities are hard to triage and patchRunning PyTorch locally hits memory and speed limits
History
steady to fading
Fine-tuning PyTorch is costly and unpredictable (2 posts)