Data and AI struggles

2,114 problems · 377 companies

MLOps

DeepSeek GPU access and drivers are painful

7 posts from 6 people5d activeFading, +0 this week

Posts per day

Posts per day7 posts, Aug 28 to Nov 21

The posts behind it

GitHub issuesvllm-project/vllm · tlrmchlsmthOct 16

[Tracking] DeepEP v2 maturation and legacy DeepEP v1 removal

Track the correctness, performance, compatibility, and migration work needed to retire vLLM’s legacy DeepEP v1 backends (`deepep_high_throughput` and `deepep_low_latency`) in favor of `deepep_v2`. **DeepEP v2 hardening** - [ ] Resolve GIN barrier data-race:...

1 reactions1 replies
GitHub issuespytorch/pytorch · rapatelOct 2

NCCL2 all_gather misclassifies zero-sized output tensors as aliased

...exception; the log does not print the pointers. ## A/B validation * Reused the exact failing container/config and changed only `TORCH_DIST_USE_NCCL2=0`; checkpoint loading passed, the 20-step proxy completed successfully. ## Expected behavior NCCL2 should handle zero-sized...

1 reactions
GitHub issuesvllm-project/vllm · LoongPeiSep 30

[Doc]: On Hopper NVSwitch nodes, TP runs can be non-reproducible even for a single request; NCCL_NVLS_ENABLE=0 alone fixes it at ~1% cost

### 📚 The doc issue The reproducibility docs (`docs/usage/reproducibility.md`, `docs/features/batch_invariance.md`) do not mention NVLS. On our machine it is the whole cause of run-to-run differences for a single request, and the cheap fix is not documented. Environment: 8×H20...

GitHub issuespytorch/pytorch · dieuctunpimpSep 30

[XPU] torch.nonzero returns empty tensor on Intel Arc B580

...tensor. Note the comparison itself works (sum is 12) - only the position materialization is broken. This breaks index-routing code (in my case MoE expert dispatch). Workaround that proves the data path is fine: move the bool mask to CPU, `nonzero` there, move indices back, then...

2 replies
GitHub issuespytorch/pytorch · dieuctunpimpSep 30

[XPU] boolean-mask indexing silently returns empty on Intel Arc B580

### 🐛 Describe the bug Related to #199157 - same workload (DeepSeek-V2-based MoE on an Intel Arc B580 with torch-xpu), same family of broken position/index handling. Filing separately since it's a different op with a different failure mode. Comparing CPU vs XPU on the same...

1 replies
GitHub issuesvllm-project/vllm · penguin-wwySep 28

[RFC]: Release all releasable GPU memory in sleep mode (tracking)

> **Update (tracking).** This issue started as a report about WorkspaceManager memory (original report below, by @penguin-wwy). Investigation showed that the workspace is one of several allocations that sleep mode leaves on the GPU. This issue now tracks all of them. ### Summary...

1 reactions2 replies
GitHub issuesvllm-project/vllm · Xuan-1998Aug 28

[Bug]: VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE declares a default of True but resolves to False

### Your current environment Not environment specific. This is a default-value mismatch in `vllm/envs.py`, reproducible on a plain checkout of `main` (`df14152`): Found while running multi-node expert-parallel serving with DeepEP v2 (`ElasticBuffer`) over NCCL GIN. ### 🐛...

1 reactions2 replies

Companies and products named

7 posts
6 posts
5 posts
4 posts
3 posts
1 post
About this problem

Evidence

7 posts from 6 people in 2 places, about 1 a week over 50 days. Mostly on vllm-project/vllm, pytorch/pytorch. Tools named alongside: NVIDIA, PyTorch, vLLM, GitHub.

Frustration Frustration 1 of 3· Seen on GitHub issues

How it was grouped

Posts that state a pain and match the "gpu-access" rule about DeepSeek. First post Aug 28, 2026, latest Oct 16, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

The same problem elsewhere

PyTorch GPU access and drivers are painful

17 posts from 17 people on GitHub issues, Reddit.

NVIDIA GPU access and drivers are painful

13 posts from 13 people on GitHub issues, Mastodon, Reddit.

GPU access, drivers and CUDA versions are painful

8 posts from 8 people on GitHub issues, Reddit.

vLLM GPU access and drivers are painful

5 posts from 3 people on GitHub issues.

Claude Code GPU access and drivers are painful

4 posts from 4 people on GitHub issues.

Ollama GPU access and drivers are painful

3 posts from 3 people on Hacker News, Reddit, GitHub issues.

OpenAI GPU access and drivers are painful

2 posts from 2 people on Reddit, GitHub issues. Unverified.

Chroma GPU access and drivers are painful

2 posts from 1 person on GitHub issues. Unverified.

Mistral GPU access and drivers are painful

2 posts from 2 people on GitHub issues. Unverified.

Other DeepSeek problems

Running DeepSeek locally hits memory and speed limits

11 posts from 11 people on GitHub issues, Hacker News.

DeepSeek users keep filing bugs

Top post: "DeepSeek-V4.1-Flash cannot run on SM120 (RTX PRO 6000): compressed-layer page_block_size=32 has no FlashInfer", one of 19 posts from 19 people on GitHub issues, Reddit.

Trouble with DeepSeek cloud

Top post: "Request: Add cloud support for DeepSeek-V4.1-Flash", one of 6 posts from 6 people on GitHub issues.

DeepSeek rate limits and usage caps block work

3 posts from 3 people on Hacker News, GitHub issues, Reddit.

Trouble with DeepSeek flash

Top post: "Asymmetric P/D Deployment for DeepSeek-V4.1 Flash", one of 3 posts from 3 people on GitHub issues, Community forums.

DeepSeek upgrades break working setups

3 posts from 3 people on GitHub issues.

DeepSeek fails to return valid structured output

3 posts from 3 people on GitHub issues.

DeepSeek runs out of context on real work

2 posts from 2 people on Reddit, GitHub issues. Unverified.

DeepSeek lock-in makes switching costly

2 posts from 2 people on GitHub issues. Unverified.

History

Momentum went from steady to fading.

Oct 23, 2026

Momentum went from fading to steady.

Oct 20, 2026

Momentum went from rising to fading.

Oct 7, 2026

First seen with 3 posts.

Oct 4, 2026