Skip to content
Problems.

No reliable way to evaluate LLM app quality

LLM apps29 posts from 28 people15d active+33 posts in the last 7 days, 3 the 7 days before (steady)

Posts per day

Posts per day29 posts, Aug 27 to Sep 25

The posts behind it

29, newest first
PostDate
qwen3.8: `think: "high"` / `"max"` silently run the default (`medium`); the model's top level `xhigh` is reachable only by a string outside the documented `think` values...three return HTTP 200, and the output is byte-identical to `"medium"`. The client has no way to tell that its level was ignored. This contradicts the thinking docs, which say a string selects "a supported level from the model's `thinking.values`". Other code paths do reject...ollama/ollamazermusSep 25today
...of curiosity. From my past experience, the latest I’ve seen were Opus-4.6 and GPT-5.5 struggle a lot with standard GHC Haskell, no extensions, nothing fancy. Not that they produced impeccable Python or Rust. But it appeared to take several turns for obvious expressions, while at...Hacker News commentsjmakerSep 25today
Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some...tcclaviger’s MXFP4 quant of Flash Next, which should make the QFN run around 2x faster.. Any suggestions for more indicative benchmarks? Anyone did a bigger comparison?. There are so many comments every week saying FP8 27B is better than low quants of QFN. I would consider...r/LocalLLaMAu/smallDeltaBigEffectSep 25today
Eval bug: VRAM usage increased from b10680 (SYCL backend)### Name and Version Docker based llama.cpp (ghcr.io/ggml-org/llama.cpp:server-intel) : Build 11011 (aa39d7a3e) with IntelLLVM 2025.3.3 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL ### Hardware AMD Ryzen 9 5900XT + 2 x Intel Arc B580 ### Models...ggml-org/llama.cppalex-rk86Sep 187 days ago
Eval bug: SYCL demands extreme scratchpad allocation (2GB+) when ngram-mod is enabled### Name and Version llama-server version: 0.4.0-dev (build 1, commit 4a89937) built with IntelLLVM 2026.1.1 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL ### Hardware Ryzen 5950X + Intel Arc A770 + AMD 6900XT ### Models Qwen3.8-27b from various sources and...ggml-org/llama.cppjintakhanSep 1312 days ago
...solutions on the cheaper ones. edit: maybe not. Astra has imitated Fable, and appears unusable for biology due to safeguards.Hacker News commentsthe__alchemistSep 1213 days ago
...un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running...Hacker News commentsAurornisSep 112 weeks ago
everything server: instructions unconditionally reference sampling/elicitation tools that only exist when the client declares those capabilities## Factual issue The `@modelcontextprotocol/server-everything` v2026.8.31 `instructions` string unconditionally tells integrating agents to use `trigger-sampling-request` and `trigger-elicitation-request`: These tools are registered through `registerConditionalTools()` inside...modelcontextprotocol/serverssasan1200Sep 112 weeks ago
...other post-training. They are highly sensitive to prompting, making sampling-based evals unreliable. Instead, we measured bits-per-byte loss on heldout data, which does not suffer from prompt sensitivity and smooths measurement of otherwise emergent abilities. As a side note, we...Hacker News commentsKaushikR2Sep 102 weeks ago
Eval bug: DSpark + LFM2.5-8B-A1B (MoE) crashes at graph_reserve — dense LFM2.5 models unaffected### Name and Version version: 0.4.0-dev (build 10853, commit 9dcf84e5a) ### Operating systems Linux ### GGML backends CPU ### Hardware ASUS Nuc 14 Pro. Intel Core Ultra Series 1 (155H) Meteor Lake. ### Models LFM2.5-8B-A1B-Q4_K_M.gguf LFM2.5-8B-A1B-DSpark-Q8_0.gguf ### Problem...ggml-org/llama.cppVaskyySep 82 weeks ago
...article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these evals,...Hacker News commentsFarmadupeSep 82 weeks ago
Amazing work! But deeply frustrating on the lack of reproducibility and how far off this is from a proper SDLC. How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere? These evals...Hacker News commentsgregwebsSep 82 weeks ago
EvaluationResult.passed asserts over non-verdict columns: dataset expectations always fail the run, and feedbacks logged without sourceRunId metadata by anyone are counted as this run's assertions...expectation whose value happens to be a bool or yes/no string is counted as a passing or failing verdict, conflating ground truth with measurements. 2. Foreign feedback. The only provenance filter in `_get_assessment_values` is `if (source_run_id := a.get("metadata",...mlflow/mlflowAUTHENSORSep 82 weeks ago
Eval bug: K2-Horizon models fail to load### Name and Version > llama-cli --version version: 0.3.0-dev (build 1, commit 95ef7fc) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware Intel(R) Core(TM) i9-14900HX NVIDIA GeForce RTX 4060 ### Models K2-Horizon-1B from...ggml-org/llama.cppRetroDev256Sep 43 weeks ago
...you mean a single minor incident involving a third party eval partner. I feel like you struggle to read. I wrote: "Intrusions by OpenAI models continued after the Hugging Face was published and acknowledged by OpenAI. They did not change their behaviour after multiple incidents,...Hacker News commentsTopfiSep 43 weeks ago
Eval bug: draft-mtp + --parallel > 1 causes cross-slot content contamination (not HIP-graphs related)## Summary With `--spec-type draft-mtp` and `--parallel N` (N > 1), concurrent requests carrying clearly distinct prompts can produce responses that drift into content belonging to a *different* concurrent request's prompt/completion. Output stays valid UTF-8 with no...ggml-org/llama.cppjsaigouSep 33 weeks ago
Eval bug: Error on loading Gemma 4 family with the new updates### Name and Version version: 0.3.0-dev (build 10744, commit 1f3d31873) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware 5070 ti+i9 14900k ### Models Gemma 4 family ### Problem description & steps to reproduce When I use MTP...ggml-org/llama.cppRyanDoesProgrammingSep 13 weeks ago
Eval bug: every mrope image decode on hybrid recurrent models (qwen3_5) triggers guaranteed non-consecutive position warnings in llama_memory_recurrent; mechanism fixed by #27941 for hybrid-idx only### Name and Version Pinned fork at commit `d6d0ce821`. Before filing we checked the relevant code at current master (`9d817213a`): the `find_slot` warning block in `src/llama-memory-recurrent.cpp` and the memory-class selection for qwen3_5 in `src/llama-model.cpp` are...ggml-org/llama.cpplloyal-researchSep 13 weeks ago
...too. Standard loops are fine, and I'll also take a paid 2-3 week trial sprint on a real bottleneck in your codebase.Hacker News commentsvarun636Sep 13 weeks ago
Eval bug: context type MTP requested but model doesn't contain MTP layers### Name and Version daef7b6874397a5a7c3d7e38b55e2ee0adf7da38 ### Operating systems Linux ### GGML backends CUDA ### Hardware H100 ### Models https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/ ### Problem description & steps to reproduce context type MTP requested but model...ggml-org/llama.cpppixel4696Aug 313 weeks ago
...and they have a ton of gaps about which models they’re doing their tests on because it’s so expensive. The idea behind any eval is so that you can pay the few pennies it costs to run an individual problem in an eval, and then as a together as a collective, we can crowdsource...Hacker News commentsljlolelAug 313 weeks ago
Claude Code for Research Papers [R]...(For me I think the eval harness and anything defining a metric should stay mine, but I keep breaking my own rule.). . Does anyone have a workflow that keeps the speedup without the detachment? Reading the diff line by line is not cutting it.. . Not looking for a “tools are just...r/MachineLearningu/NeatFox5866Aug 303 weeks ago
Eval bug: llama-server crashes (stack overflow / access violation) on hybrid recurrent models (qwen3_5_moe) with mmproj vision when alternating text and image turns### Name and Version Also reproduced on b10639 and on current master sources (checked `src/llama-memory-recurrent.cpp` and `tools/server/server-context.cpp` - identical in master). ### Operating systems Windows ### GGML backends CUDA, CPU ### Hardware Ryzen 9 9950X3D, 96 GB RAM,...ggml-org/llama.cppIvanAndreychikovAug 293 weeks ago
...to perform suspiciously well in public benchmarks vs private ones and what models still struggle with along with why, so yeah, certainly can always improve but I got, I'd argue, enough of an idea to where my critique of LLM coding limitations has legs. Which brings us to what I...Hacker News commentsTopfiAug 293 weeks ago
Eval bug: Dflash2: vision support is still broken...testing in with qwen 3.8 in llama.cpp webui, I found out that vision support (with PDFs) is broken, while the native mtp is not. checkpoint from https://huggingface.co/incoai/Qwen3.8-27B-DFlash2-GGUF parameters (as suggested): model-draft = Qwen3.8-27B-DFlash2-Q4_K_M.gguf...ggml-org/llama.cppshowgood163Aug 284 weeks ago
Eval bug: SYCL multi-GPU crash with Intel Arc Pro B50 + Arc A770### Name and Version llama.cpp build: b10644-d7a207411 The executable was built locally from the ggml-org/llama.cpp repository. version: 0.3.0-dev (build 10644, commit d7a207411) built with IntelLLVM 2026.1.0 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL...ggml-org/llama.cppnicolataibiAug 284 weeks ago
Eval bug: DFlash2: --split-mode tensor aborts with SPLIT_AXIS_UNKNOWN on ROCm (Qwen3.8-27B, 2x RX 7900 XTX)### Name and Version Current master: b10660 6c84c7d5d model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) This includes the merged DFlash2 implementation from #27342. ### Operating systems Linux ### GGML backends HIP ### Hardware Ryzen 3600 Asus prime x570 pro (8x/8x) 2 x rx 7900...ggml-org/llama.cppa-n-t-0Aug 274 weeks ago
Eval bug: crash on RPC server when using gemma-4-26B MTP head### Name and Version (base) MacStudio1:~ alvise$ /opt/llama.cpp/llama-server --version version: 0.1.0-dev (build 10442, commit 9b0a2ce85) built with AppleClang 21.0.0.21000101 for Darwin arm64 ### Operating systems Mac ### GGML backends Metal ### Hardware 2x Mac Studio M4 Max...ggml-org/llama.cppdorigoaAug 274 weeks ago
[DOC] Clarify that Detect Issues analyzes trace data only, not source code> [!WARNING] > Before submitting a PR, please make sure that: > - A maintainer has triaged this issue and applied the `ready` label > - This issue has no assignee > - No duplicate PR exists > > PRs not meeting these requirements may be automatically closed. ### Summary The...mlflow/mlflowadamguraryAug 274 weeks ago

Companies and products named

Company or productPosts naming it
llama.cpp13
NVIDIA7
Hugging Face6
Qwen4
GitHub3
MLflow2
About this problem

Evidence

29 posts from 28 people in 7 places, about 6 a week over 30 days. Mostly on ggml-org/llama.cpp, Hacker News comments, mlflow/mlflow. Tools named alongside: llama.cpp, NVIDIA, Hugging Face, Qwen.

Frustration Frustration 1 of 3· Seen on GitHub issues, Hacker News, Reddit

How it was grouped

Posts that state a pain and match the "evals" rule. First post Aug 27, 2026, latest Sep 25, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

History

  • 2026-09-25 Added: No reliable way to evaluate LLM app quality (26 posts)

Rising problems by email

Mondays: the problems in data, tech and AI that grew fastest that week.

Double opt-in. Unsubscribe any time.