Problemsby GlyphLLM token costs are hard to predict and control
14 posts from 14 people12d activeFading, +0 this week
Posts per day
The posts behind it
I built a scikit-learn for RSI [P]
...I keep coming back to familiar problems from RL and evolutionary search:. Sampling: How do we find promising changes efficiently?. Evaluation: How do we know a new variant actually improves the agent?. Diversity: How do we avoid converging too early?. That motivated us to build...
[Bug]: Token spend/budget limit not enforced for keys created or used for Admin users level
...(max_budget / tpm_limit / token limit) on an API key created for an admin user, LiteLLM fails to enforce the limit once the budget is exhausted. Admin users can continue making AI requests past the defined maximum amount without receiving any budget exhaustion error (e.g., HTTP...
How should we evaluate AI Agent observability platforms in 2026?
Disclosure: I work at Bonree, the company behind Bonree ONE, an AI observability platform. I’m asking about the general problem of choosing a platform, and I’m not trying to pitch ours.. AI agents are moving from chat interfaces into production workflows, and teams are now...
Bug: torch.linalg.qr not implemented on Apple Silicon MPS — Heretic 1.4.0 crashes
### 🐛 Describe the bug # Bug: `torch.linalg.qr` not implemented on Apple Silicon MPS - Heretic 1.4.0 crashes ## Description When running Heretic 1.4.0 with `Qwen/Qwen3-4B-Instruct-2507` on an Apple M3 MacBook Pro with 16 GB of unified memory, the process crashes during model...
How should I implement long-term memory for an AI character chat app without sending the entire conversation history to the LLM?
I’m building a character-based chat application with Next.js and PostgreSQL.. Each character has relatively static data such as:. personality. . backstory. . speaking style. . scenario. . character-specific instructions. . But I also want characters to remember information...
feat: option to limit the skill manifest to the skills attached to a model
### Before Submitting - [x] I searched open and closed issues and discussions for an existing report. - [x] I reproduced this bug on the latest release AND on the current `dev` branch, right before submitting this report. I did not just check an old version or rely on a check...
[P] Open-Sourcing a sub-15ms Deterministic Decision Gateway to eliminate 2-second LLM routing latencies (FastAPI/Python)
...decisions (e.g., safety guardrails or API triage). Token generation is inherently too slow and expensive for operational routing.. To solve this at scale, I built and open-sourced the "Dual-Process Jev Gateway.". It places a sub-15ms deterministic decision layer (System 1) in...
[PERF] DeepSeek-V3.2 / GLM-5 lightning indexer overallocates memory
### System Info * `transformers` version: `5.19.0.dev0` * Platform: `Linux-6.8.0-1069-gcp-x86_64-with-glibc2.35` * Python version: `3.10.12` * `huggingface_hub` version: `1.31.0` * `safetensors` version: `0.8.0` * `accelerate` version: `1.14.0` * PyTorch version (accelerator?):...
..."But inference costs play a massive role here too, which I think is still the biggest bottleneck in AI development. Running high-impact models for math, medicine, or programming requires a ton of compute. Throw in the politicization of everything AI-related, and it feels like...
Teams that moved from LLM APIs to self-hosting: was it worth it?
...cost once you add up GPUs, idle time, and the engineering hours?. what was the most painful part? cold starts, autoscaling, OOMs, model quality, getting paged at 2am. if you went back to APIs, what made you switch back. ballpark numbers are totally fine. I'll put together a cost...
[Performance][ROCm][gfx1100] compressed-tensors silently enables fp8 KV cache, which is far slower than bf16 for paged decode attention on RDNA3
### Summary On a Radeon RX 7900 XTX (gfx1100, RDNA3), a compressed-tensors checkpoint that carries a `kv_cache_scheme` gets its KV cache silently switched to `fp8`, even though `--kv-cache-dtype` is left at its default `auto`. In this configuration decode attention is...
Add prompt caching support for Mistral gateway endpoints
...reduces our Mistral API spend and improves end-user latency. #### Why is it currently difficult to achieve this use case? The MLflow AI Gateway does not expose Mistral's `prompt_cache_key` parameter. Users cannot enable prompt caching without bypassing the gateway and calling...
Companies and products named
About this problem
Evidence
14 posts from 14 people in 11 places, about 2 a week over 57 days. Mostly on Hacker News comments, r/mlops, vllm-project/vllm. Tools named alongside: NVIDIA, PyTorch, GitHub, vLLM.
Frustration Frustration 0 of 3· Seen on GitHub issues, Reddit, Hacker News, Stack Overflow
How it was grouped
Posts that state a pain and match the "llm-cost" rule. First post Sep 5, 2026, latest Oct 31, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Anthropic token costs are hard to predict
6 posts from 6 people on Hacker News, GitHub issues, Reddit.
Claude token costs are hard to predict
5 posts from 5 people on Hacker News, App Store reviews, Reddit.
OpenAI token costs are hard to predict
3 posts from 2 people on GitHub issues, Hacker News.
Claude Code token costs are hard to predict
2 posts from 2 people on Reddit, Hacker News. Unverified.
LlamaIndex token costs are hard to predict
2 posts from 2 people on GitHub issues. Unverified.
Gemini token costs are hard to predict
2 posts from 2 people on Reddit, GitHub issues. Unverified.
History