Problemsby GlyphNo reliable way to evaluate LLM app quality
153 posts from 144 people53d activeSteady, +11 this week
Posts per day
The posts behind it
Eval bug: llama-server hangs indefinitely when loading a ~19 GB model split across TWO RPC backends (single-backend loads work)
### Name and Version - llama.cpp `0.9.0-dev`, commit `10d4846a` (2026-11-05), identical build on all three machines - Host: Raspberry Pi 5, 16 GB RAM, Debian 13, model file on USB SSD - Backend A: Mac mini (Apple M1, 8 GB), Metal, `ggml-rpc-server` - Backend B: Raspberry Pi 5, 8...
Eval bug: CPU F16 dot product accumulates in F16 on ARM (FP16 arithmetic, no SVE), overflows to inf
...the issue): I built it with (clean master, 13ec8f614, Apple c - not homebrew which had a problem with accelerate headers) Output: This code shows the dot product being calculated wrongly by ggml. It takes a row of F16 with a column of F32 (x 64, all positive - mixed signs cancel...
trainlint: static checks for ML bugs that don't crash (fit-before-split, eval mode, logits losses) [P]
...so it needs no. data, model or GPU.. How well it works, with the caveats. I wrote 20 buggy scripts with fixed twins, but I wrote them. together with the rules, so they only protect against regressions. For an outside check I read. every finding on pytorch/examples at a pinned...
Anyone else tired of how expensive agent evaluations are? [D]
...environments takes so much time and compute and then reproducing results is another headache. My goal is to test which VLMs can be better in general for agents irrespective of a singular domain.. I found there’s a benchmark called AgentVQA that tests how well models understand...
Eval bug: --moe-cache-mib aborts at load in llama_moe_cache::impl::copy on ROCm/HIP (RX 6800 XT, gfx1030, Qwen3.6-35B-A3B) — cache size independent
## Environment - **GPU**: AMD Radeon RX 6800 XT (gfx1030), PCIe 4.0 x16 - **ROCm**: 7.2.0 (system install at /opt/rocm) - **llama.cpp**: b12110 (tag), built from source: `-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release` - **OS**: Ubuntu (Linux x86_64) -...
get_optimizer_state_dict and set_optimizer_state_dict raise KeyError after a no-grad forward on a fully_shard model
### 🐛 Describe the bug We hit this saving optimizer state after an eval pass in Halo (github.com/whitecircle/halo). We now reshard before every read, and I'd like to drop that. #### AI-generated details (Claude Code), reviewed by me After a forward with no backward,...
What should a benchmark of agent direction do when the assignment itself is broken?
Full transparency up front: I am the founder of HyperHat, and this post comes out of what we are building. No link in here.. When we posted our first benchmark results here, the pushback that stuck with me was about judgment, not execution. The example: two requirements that...
SQL Expressions x Assistant
Grafana Assistant currently relies on sandboxes to answer ad-hoc questions requiring data transformation. How might a SQL Expressions tool stack up against the Python sandbox approach? To find out, we will: * Create a SQL Expressions tool in Grafana Assistant * Run local tests...
Eval bug: CUDA error when loading DFlash with sm tensor
### Name and Version $ ./llama-cli --version version: 0.8.0-dev (build 12080, commit fbb9148ee) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware 2x RTX 3090 with NVLink ### Models Muse Glimmer (Q8_K_XL from...
Eval bug: granite_swa: per-layer RoPE theta (layer_rope_theta) is ignored
### Name and Version version: 0.8.0 (build 11960, commit 8642541b2) built with AppleClang 21.0.0.21000334 for Darwin arm64 ### Operating systems Mac ### GGML backends Metal ### Hardware Apple m5 max, 64GB ram ### Models no released checkpoint is affected, but any future...
What actually broke when we put agents on back-office work
...corrections become the eval set.. Credentials. One shared API token for "the bot" was a mess. When something went wrong we could not revoke one agent without killing the rest. Each agent gets its own scoped token, a named owner, and an expiry.. Curious what broke for you once...
Eval bug: SYCL: decode hangs in ggml_backend_sycl_synchronize on Xe3 (Panther Lake) unless GGML_SYCL_USE_ASYNC_MEM_OP=0
...Qwen3-4B Q4_K_M (Qwen3-4B-GGUF). Other models on this build were run only with the workaround below. ### Problem description & steps to reproduce With default environment settings, generation hangs forever on the first decode token. Prompt processing completes normally; then one...
Eval bug: Qwen 2.5 coder doesn't use tools
### Name and Version version: 0.8.0-dev (build 12030, commit a91eab67d) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan ### Hardware Vulkan0: AMD Radeon Graphics (RADV RENOIR) (8954 MiB, 7977 MiB free) CPU: AMD Ryzen 7 5825U with...
PERF: pd.eval() copies the whole result in reconstruct_object
### Pandas version checks - [x] I have checked that this issue has not already been reported. - [x] I have confirmed this issue exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas. - [ ] I have confirmed this issue exists on the main...
Switch node has new timezone conversion error that is embedded in the node.
### Bug Description Workflow cannot be activated because required inputs are not connected: the inputs of 'Merge' could not be determined (Expression evaluation failed: Invalid timezone: "America/Dallas"). No "timezone" variable data is passed by the UI/user. ### To Reproduce...
Expr.get() raises ComputeError: expected single index inside list.eval()
### Checks - [x] I have checked that this issue has not already been reported. - [x] I have confirmed this bug exists on the [latest version](https://pypi.org/project/polars/) of Polars. ### Reproducible example ### Log output ### Issue description Expr.get() supports vectorized...
...access I eval them on my home assistant set, a small coder benchmark, etc. and they are too expensive, and too low quality. The benefit for Google is their chat interface has privileged access to their data (maps, etc.) but their model itself is low quality and, if I'm being...
[FR] Record who created a judge/scorer and who enabled online scoring (created_by), and expose it via CLI/MCP
...or organization? We run many online evals across a large organization and have no way to attribute cost or review a judge's setup with its owner. > Why is it currently difficult to achieve this use case? `Scorer` stores only `experiment_id`, `scorer_name`, `scorer_version`,...
Copilot sanity tests: CAPI rejects pipeline test token (org access)
...this secret (#323667). Could one of you check the test account behind it? ## What's failing Since Oct 28 around 14:00 UTC, every `Run Copilot sanity tests` step in the product build fails on all 3 platforms (Linux x64, Windows x64, macOS arm64). All 3 tests fail on the first...
Eval bug: glm-ocr GGUF (Ollama conversion) has no chat_template — default framing causes template-token repetition loops; pass-through template fixes it
...makes the failure look intermittent - content- and image-size-dependent. **Verified workaround:** a pass-through chat template fixes it completely: With this, previously-looping pages transcribe correctly (e.g. an ornament type-specimen page that looped under the default...
CPT’d Qwen9B on Polish legal data. How to make it chat? IMPORTANTLY: how to make it reason? and function call in my harness? and which to do first? [D]
...an agentic+chat assistant that reasons through legal problems. . But more importantly, how do I get reasoning, teach my in-harness tool-use training, and where does SFT vs RL belong in that process and in what way (especially confused about reasoning and SFT)? . What I’ve...
Eval bug: Vulkan multi-GPU layer split on 2x Intel Arc Pro B65 (Battlemage, Mesa 26.2.5 ANV): decode drops ~35% and becomes unstable, degrades over repetitions
### Name and Version version: 0.7.0-dev (build 1, commit c7fc509) = tag b11804, built from source with -DGGML_VULKAN=ON -DGGML_NATIVE=ON (GNU 16.2.1), in an Arch Linux container with Mesa 26.2.5-arch1.1 (vulkan-intel, Vulkan API 1.4.354). Host: Ubuntu 26.04, kernel...
Eval bug: GGML_VK_PERF_LOGGER=1 aborts with two Vulkan devices: GGML_ASSERT(ctx->compute_ctx.expired()) at ggml-vulkan.cpp:14341
### Name and Version version: 0.7.0-dev (build 1, commit c7fc509) = tag b11804, -DGGML_VULKAN=ON, Mesa 26.2.5 (ANV), Linux 7.0.0-38 (xe). ### Operating systems Linux ### GGML backends Vulkan ### Hardware 2x Intel Arc Pro B65 (BMG G31), Xeon W-2295. ### Problem description &...
Eval bug: Stream resume produced no new bytes, giving up Llama.3.2.1Bl on llama-server returns error
### Name and Version version: 0.7.0-dev (build 11913, commit 7aca0d2ea) ### Operating systems Windows ### GGML backends CPU ### Hardware Running on HP notebook G85, 8GB Ram, 500 GB SSD ### Models Llama-3.2-1B-Instruct-Q4_K_M.gguf ### Problem description & steps to reproduce Eval...
Eval bug: GGML_ASSERT(t_start > 0) in server-common.h set_prompt_last aborts llama-server on Windows (0xc0000409) after ~1.5M tasks
...appears in two crashes, on 2026-09-29 09:51:33 and 2026-10-27 10:26:11 (local time). The failing check is in `server_slot`'s timing helper, `set_prompt_last()`: `t_start` is set only by `update_prompt_start()`. On f0c41e016, `update_prompt_last()` is called without a `t_start`...
Eval bug: GSQ-RCO implementation of DSV4.1 does not load
...### Problem description & steps to reproduce The model fails to load instantly. I don't believe this is due to insufficient hardware; Unsloth's much larger IQ3_S quant of this model loads fine, but is very slow afterwards. Here are my parameters: ### First Bad Commit _No...
Eval bug: GLM 5.3 (non flash) doesn't load with Tensor Split and CUDA backend
### Name and Version version: 0.7.0-dev (build 1, commit a919179) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware G7.24xlarge AWS instance 92 Xeon vcpus + 4 * RTX 4500 Pro ### Models...
We built powerful reasoning harnesses, only to wrap them in rigid workflows. What if we just childproofed the harness instead?
...harness is the application: Let the model plan and run the execution loop. Hard-coding workarounds for today's model limitations just bakes in tech debt for when tomorrow's model natively bypasses them anyway.. English changes the system: Splitting the "use case engine" from a...
`Trainer.evaluate` eval_loss under DDP includes the samples the sampler repeats to even out the last batch
Hi! I was comparing `Trainer.evaluate` on 2 processes with one process evaluating the same global batches (per-device batch twice as large), and the `eval_loss` can differ when the eval set is not a multiple of the global batch, on 5.20.2 and 4.57.6. If I read `evaluation_loop`...
Eval bug: qwen4exp assert with merged tensors from #31028
### Name and Version version: 0.7.0-dev (build 4104, commit d854d98ba) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan ### Hardware AMD Ryzen™ AI Max+ 395 ### Models BF16 converted:...
Eval bug: Segmentation fault during prefill with mixed ROCm/Vulkan
### Name and Version llama-cli --version version: 0.6.0-dev (build 11429, commit d812350493) built with GNU 16.2.1 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan ### Hardware MI50 using ROCm and 7900 XTX using Vulkan. ### Models...
Eval bug: mtmd-cli reuses a stale mtmd batch after /clear: previous image's embeddings returned, or GGML_ASSERT(offset <= batch->output_embd.size())
### Name and Version llama.cpp master 63412d8737b3efa30748dd79ceb2c93487e19f2b (2026-10-25). Also seen on release b11860 (df9beac7153166550571c70cc667f270b745da49), where the assert is at `mtmd.cpp:2413`. Build: ### Operating systems Mac ### GGML backends Metal ### Hardware...
/v1/completions applies the chat template to raw prompts
...`/api/generate` with `raw: true` does the same in Ollama, but `/v1/completions` offers no way to turn templating off. `FromCompleteRequest` in `openai/openai.go` builds the `api.GenerateRequest` without setting `Raw`, so every completion request goes through the template. The...
Eval bug: mtmd: server crash: `GGML_ASSERT` when encoding large images (`--image-max-tokens` not respected)
### Name and Version `74bca3950` `(HEAD -> master, tag: b11870, ggml-org/master, ggml-org/HEAD)` ### Operating systems Linux ### GGML backends CUDA ### Hardware NVIDIA GeForce RTX 4060 Ti (15947 MiB, 15678 MiB free) AMD Ryzen 7 7700X 8-Core Processor (127916 MiB, 127916 MiB...
Eval bug: [SYCL] oneDNN fused-SDPA flash-attention path only used for first context in a process; subsequent contexts fall back to native kernel
### Name and Version Commit 7894e8b92 (11852), Ubuntu 26.04.1, Core Ultra 5 125H (Meteor Lake Arc iGPU), oneAPI Intel(R) oneAPI DPC++/C++ Compiler 2026.2.0 (2026.2.0.20261001) ### Operating systems Linux ### GGML backends SYCL ### Hardware Intel(R) Core(TM) Ultra 5 125H (18) @...
Eval bug: OpenVINO backend (GPU, Windows): process memory grows ~35 MB per request and never levels off. Reproduces on b11869
### Name and Version b11869 (d327bf1f0), also v0.7.0 (620d2225c). Built from source as a library; we didn't use llama-cli. ### Operating systems Windows ### GGML backends OpenVINO ### Hardware Intel Core Ultra 7 365, Intel Arc integrated GPU (driver 32.0.101.8826, also 9150), 32...
Eval bug: Vulkan: count_experts.comp zero_skipped_rows() loops over every id in every thread
https://github.com/ggml-org/llama.cpp/blob/6050b467c07a95c7b87debd4ddac2f8ba8d9b807/ggml/src/ggml-vulkan/vulkan-shaders/count_experts.comp#L59 In count_experts.comp, the `zero_skipped_rows()` loop (from #26631) runs over every expert id in every thread instead of each thread...
SIGSEGV in MatchImpl::vectorConstant on LowCardinality column pruned by query_plan_remove_unused_columns + PREWHERE partial-header eval
`MatchImpl::vectorConstant` SIGSEGVs when `query_plan_remove_unused_columns` (default ON) prunes a column that is only used in `WHERE`/`PREWHERE` whose value is `LowCardinality(String)`, and the subsequent `transformHeader → updateHeader → evaluatePartialResult` ends up...
Eval bug: ggml-hexagon: vision encoder (mmproj/CLIP) output corrupted depending on image shape on HTP v75 — garbled transcription with repetition loop, correct on CPU
### Name and Version Official prebuilt release `llama-b11849-bin-android-arm64-snapdragon.tar.gz` (also reproduced with a self-built b11849, NDK r30 + Hexagon SDK 6.7): ### Operating Systems Android 15 (HyperOS 2), unrooted, app process and adb shell both reproduce. ### GGML...
Eval bug: Nemotron Vulkan Intel Core Ultra 5 125H
### Name and Version tim@minisforum-ai:~/llama.cpp$ ./build/bin/llama-cli --version version: 0.7.0-dev (build 11852, commit 7894e8b92) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan ### Hardware Minisforum M1 Pro-125H: Intel Core...
Eval bug: Gemma 4 31B + draft-mtp: CUDA illegal memory access when MTP drafts right after the KV cache is rolled back or cleared (checkpoint restore or full re-process); no crash with MTP off
...right after a **full prompt re-processing** ("forcing full prompt re-processing due to lack of cache data ... memory_seq_rm [0, end)"), 0.6-2 s into generation, again only with MTP. Workload: short (~2.3k-token) prompts with `json_schema`, n_predict 700, each read from scratch....
[Chat Parsing] Add python-call-list content parser and strip_prefix/strip_suffix text args
...` … ` in thinking mode only, so the wrapper has to be optional. An `open`/`close` pair doesn't work: `content` is the fallback field, which has no `open`, and the answer has no tags when thinking is off. ### Motivation In Transformers.js we are integrating response parsing:...
[Feature][ROCm][DSv4.1] Mono decode: enable MI350P (gfx950, 128 CUs) via k-pass task decomposition
### 🚀 The feature, motivation and pitch **Goal:** run the DeepSeek-V4.1 mono decode kernels on **MI350P** (gfx950, **128 CUs**, 4 XCDs). The mono decode kernels (`vllm/models/deepseek_v41/amd/mono/`) place **one resident CTA per compute unit**, so the grid width `BLOCKS` must...
How do you audit AI agent decisions in regulated workflows?
...they still agree with humans as things change?. . I'm trying to learn what's really painful versus what only sounds painful. Comments or DMs both welcome, and I'll post a summary of what I learn..
Eval bug: qwen4exp + --spec-type draft-mtp aborts with GGML_ASSERT(buffer) in llm_graph_input_kpool::set_input on server start
### Name and Version Also reproduced on b11491 (`9b4ed0ca5`). By code inspection the affected code is unchanged on current master `dd002b91f` (2026-10-20); the fix below applies cleanly there. ### Operating systems Linux (Ubuntu 24.04, kernel 6.17) ### GGML backends Vulkan ###...
Eval bug: SYCL F16: NaN output on dense Qwen3.8-27B Q4_K_M since #29107 (IQ3 code reorder); fused GEMM off fixes it (Arc Pro B70)
...on the same build, so the test shapes don't reach this path (they're small; the failing GEMMs here are K=5120, M up to 248320). **Regression:** same model, same flags, `-DGGML_SYCL_F16=ON`, each tag built fresh: | tag | result | |---|---| | b11491, b11493 (#29245 adds...
My mutation test run killed 252 of 252 mutants. Every kill was fake.
...a file the copy is missing.. . "Killed" isn't evidence on its own. The log names the failing tests for each kill. If one test name shows up in every kill, thats not a strong suite, it's one broken test.. . When a test starts reading a new file, the file goes on the copy list in...
SteerTTS: 23M-param English TTS you steer with numbers (pitch, rate, loudness, etc. as z-scores), runs real-time on CPU [P]
...English sets (run locally, not an official submission, have reached out for it).. What doesn’t work well (we tried to be upfront): . Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak. Emotion tags behave as style switches; an independent...
How do I make AI actually think and execute instead of constantly babysitting it?
I'm an ML engineer and heavy AI user. My biggest frustration isn't model intelligence-it's having to manage the AI's thinking.. I constantly have to:. Ask it to write a better prompt, then paste that prompt back.. Remind it to research, think critically, compare alternatives,...
Eval bug: Gemma 4 thinking control not working via OpenAI API
### Name and Version version: 0.7.0-dev (build 11715, commit 9d1878a43) built with GNU 14.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan, CPU ### Hardware RX 7600 and Ryzen 7 7700 ### Models Gemma 4 E4B and E2B from the official GGML huggingface repo,...
[Bug] Claude keeps writing bash syntax under zsh: 51 failures per 1,000 Bash calls, 76% silent (follow-up to #67146)
...it's installed.** `CLAUDE_CODE_SHELL=/bin/bash` (from #74746) works, but it's opt-in and hard to find. The tool is called "Bash", and the model writes for bash. 3. **Report zsh errors as failures.** When a result contains `(eval):N:` shell errors, tell the model the call failed...
Eval bug: Mellum 8x slower on CUDA since #30584 (ffn_moe_down MUL_MAT_ID falls back to CPU)
### Name and Version ### Operating systems Linux ### GGML backends CUDA ### Hardware NVIDIA GeForce RTX 3050 6 GB (compute capability 8.6), 16 GB RAM, CUDA 13.0 ### Models https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF (commit `20b4394`), `MXFP4_MOE` and...
Eval bug: --moe-cache-mib crashes at load on Vulkan (RX 6900 XT, Windows)
### Name and Version version: 0.7.0-dev (build 11700, commit 23f7aabc1) built with Clang 20.1.8 for Windows x86_64 Official Windows x64 Vulkan release zip ### Operating systems Windows ### GGML backends Vulkan ### Hardware GPU: AMD Radeon RX 6900 XT 16 GB (gfx1030), AMD...
Eval bug: Regression in #30257: Gemma 4 with --mmproj and MTP (-md) aborts on startup with GGML_ASSERT(ubatch->embd_state)
### Name and Version llama-server version: 0.7.0-dev (build 11700, commit 23f7aabc1) ### Operating systems Linux ### GGML backends HIP ### Hardware AMD Ryzen 7 9700X, 2x AMD Radeon AI PRO R9700 (2x 32 GB) on ROCm 10.1. ### Models Main: Gemma-4-31B-it (Q8_0) Draft:...
Eval bug: Beam-1.0-501B-A23B-GGUF outputs within thinking block
### Name and Version version: 0.7.0-dev (build 11705, commit ea0c4c2c8) built with GNU 14.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware AMD Ryzen 9 7900X Nvidia RTX Pro 6000 Blackwell ### Models Reflection Beam 1.0 501B A23B [Unsloth GGUF...
[Truffle] NoClassDefFoundError escapes after OptimizedCallTarget$ReturnProfile class initialization fails at stack exhaustion
## Describe the issue On the current master a plain JavaScript program can kill the launcher with a host `NoClassDefFoundError`. When the guest stack is nearly exhausted, the exception-profiling path added by GR-80571 (`OptimizedCallTarget.handleException` ->...
Expose reasoning (thinking) token counts in API usage for thinking models
...were spent on reasoning. `completion_tokens` appears to include them, but there is no way to separate "visible answer" tokens from "thinking" tokens. ### Current behavior (observed 2026-10-18, ollama.com) OpenAI-compatible endpoint `POST https://ollama.com/v1/chat/completions`:...
I turned the Page Objects from my Playwright tests into tools my coding agent can call on the live page
...tokens. Your e2e tests already know exactly how your application works, the agents just can't make any use of them.. I just published a small playground app to get some feedback before locking in the alpha release :). Playground App, nothing to install: ayme.dev/playground ....
Eval bug: mellum2/vulkan: token generation performance dropped from 44t/s to 7.8t/s
### Name and Version ./llama-server -m ~/models/Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf --parallel 1 --ctx-size 16384 --predict 16384 --ubatch-size 512 --temp 0.6 --top-p 0.95 --top-k 20 --cache-type-k q8_0 --cache-type-v q8_0 b11696: 7.8t/s b11693: 7.8t/s b11685: 7.8t/s b11681:...
Eval bug: Vulkan draft-mtp decode ~4x slower and prefill ~6x slower since #30020 (target compute buffer re-allocated into GTT)
### Name and Version v0.7.0 (620d2225c, b11669), Vulkan backend, built `-DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release`. The last good build tested is v0.6.0 (d81235049). ### Environment - 2x AMD Radeon AI PRO R9700 32 GB (gfx1201, RDNA4). One card is visible to...
Eval bug: qwen4exp MTP — draft-only export (null trunk tensors) segfaults during context reservation
# qwen4exp: MTP draft loaded without a trunk segfaults during context reservation **Component:** llama.cpp (src/models/qwen4exp.cpp) - speculative MTP (`--spec-type draft-mtp`) **Type:** crash / unguarded null dereference ## Summary Loading a qwen4exp (Qwen3.8-Flash-Next) MTP...
Eval bug: Nemotron 3.5 Bolt 74B-A9B segfaults on Metal in ggml_metal_op_mul_mat_id
### Name and Version Built from source (`cmake -B build -DCMAKE_BUILD_TYPE=Release`), includes #30520. ### Operating systems Mac (macOS 27.0, 26A428) ### GGML backends Metal ### Hardware Apple M1 Max, 32 GB (32-core GPU, GPU family 7) ### Models...
Eval bug: crash while loading clef-flash on arm64 CPU
### Name and Version $ ./llama-server --version version: 0.6.0-dev (build 11643, commit 1ddc26255) built with GNU 14.2.0 for Linux aarch64 ### Operating systems Linux ### GGML backends CPU ### Hardware NVIDIA DGX Spark ### Models Clef Flash GGUF Q4_K_M ### Problem description &...
Eval bug: rpc-server -c: cache files truncated on disk-full are later served as valid cache hits → silent weight corruption / garbage output
### Name and Version version 0.6.0-dev (build 968, commit 10a60cf ### Operating systems Linux ### GGML backends CUDA, RPC ### Hardware V100 in a KVM guest as the RPC server, client with RTX 5070 Ti ### Models Qwen3.8-27B Q8_0 / UD-Q8_K_XL ### Problem description & steps to...
Eval bug: convert_hf_to_gguf: Granite Switch conversion writes block_count one too high for newer checkpoints
...the patch, `ibm-granite/granite-switch-4.1-3b-preview` converts but `llama-server` fails to load it with the error above. - With the patch, the same checkpoint converts and `llama-server` reports `model loaded`. - An earlier SingleSwitch-generation revision of the same repo:...
Eval bug: Mellum NaN on Metal with Q8_0 (ffn_moe_down input exceeds F16 range)
### Name and Version $./llama-cli --version version: 0.6.0-dev (build 11626, commit df18bee11) built with AppleClang 17.0.0.17000013 for Darwin arm64 ### Operating systems Mac ### GGML backends Metal ### Hardware Apple M2 Max, 64 GB ### Models...
[RFC]: Bring-Your-Own-GPU (BYOG) reproducible evals and benchmarks for PRs
### Motivation. Reviewing performance- and accuracy-sensitive PRs is frequently bottlenecked on evidence and regression testing. Reviewers routinely ask "can you please provide e2e performance data?", "can you please provide eval results?", or "can you show 8k1k numbers on...
Our PM wants prompt edit access, which parts are safe to hand over
...it explains our pricing. she knows that better than any of us, and eng reviewing it is a bottleneck with no upside imo. the rest behaves like code and the tool descriptions decide which tool the model picks on ambiguous requests, there’s an output format a parser depends on, a...
[BUG] genai.evaluate(predict_fn=...) drops scorer assessments when the agent trace is started off-thread
> [!WARNING] > Before submitting a PR, please make sure that: > - A maintainer has triaged this issue and applied the `ready` label > - This issue has no assignee > - No duplicate PR exists > > PRs not meeting these requirements may be automatically closed. ### Issues Policy...
HOLIDAY() time range returns 500 instead of 400 when given a country
...States"`, `"United Kingdom"` and `"England"` all 500, which makes the accepted set hard to guess. ### Reproduction 1. Explore any chart with a temporal column. 2. Open the Time Range control and switch to Advanced. 3. Put `HOLIDAY("Christmas Day", "UK")` in START and `now` in...
Always start PR work in a new git worktree (AGENTS.md + pr-flow)
## Problem Nothing in `AGENTS.md` or the `pr-flow` skill requires work to start in a git worktree. `AGENTS.md` → *Issue-driven work style* says to create the `v2/ / - ` branch from `origin/v2/main`, and `pr-flow` step 2 does it with `git switch -c` in whatever checkout the...
I built a pytorch library of pretrained models for 3D point clouds following timm standards [P]
...on during the past few years during my free time. I started this project after being unable to use torch-points3d at the time (too many installation errors). . Note that this project is still in active development and in alpha. I am looking for early users and feedback! . What...
OLLAMA_NUM_PARALLEL is silently forced to 1 for hybrid architectures (qwen35moe and others); one slot means one prompt cache for every client
...is on and how many slots were requested. PR follows. If there is a known correctness problem with several sequences on a specific family in the llama.cpp runner, a per-family list would be better than the blanket pin; I could not find one for `qwen35moe` after a day of four-slot...
How to get both losses and detection predictions from Torchvision Faster R-CNN in training mode and vice versa?
I am using PyTorch's torchvision.models.detection.fasterrcnn_resnet50_fpn_v2 to train an object detection model.. By default, when I call the model in training mode, it returns a dictionary containing the losses. When I switch to evaluation mode, it returns a list of predictions...
Eval bug: qwen4exp (Qwen3.8-Flash-Next) + MTP aborts on image input after #30257 (`GGML_ASSERT(ubatch.token …)` left in `graph_mtp`)
...Possibly related #30032 (qwen4exp QSA indexer OOM when vision is active) is a different problem with the same model family and vision. ### Attachment `qwen4exp-remove-assert.patch`:
Eval bug: GLM-5.3-Flash + mmproj + --spec-type draft-mtp: every image request fails with "failed to process speculative batch" (Vulkan), no per-request opt-out
...request fails the same way, so a client cannot work around the bug per request; the only workaround is to run the whole server without `draft-mtp`, which costs the speedup for all text traffic. 2. The server survives (the slot is cancelled, later text requests are fine), so it...
docs(core): fix grammatical phrasing in EvaluatorCallbackHandler docstring
### Description In libs/core/langchain_core/tracers/evaluation.py, the docstring for EvaluatorCallbackHandler.project_name and __init__ contains a minor grammatical typo: - Current: The LangSmith project name to be organize eval chain runs under. - Expected: The LangSmith...
Eval bug: Mistral Small 4 CUDA expert down-projections fall back to CPU despite 37/37 layers offloaded
### Name and Version Upstream ggml-org/llama.cpp, commit: c811cb8f0ac91b8ac72a32f970bdd45037f20da7 llama-server --version: version: 0.6.0-dev (build 11507, commit c811cb8f0) built with GNU 13.3.0 for Linux x86_64 CUDA container: 13.3.0 Build configuration:...
Eval bug: SYCL: performance regression on PP
### Name and Version root@40f8a8a6c4d7:/app# ./llama-cli --version version: 0.6.0-dev (build 11514, commit de7fa0a3c) built with IntelLLVM 2026.1.1 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL ### Hardware B580 + 7 5700X3D + 48GB at 3133MHz ### Models...
Eval bug: EmbeddingGemma-2 vision is generating more tokens than set.
### Name and Version Using `ghcr.io/ggml-org/llama.cpp:server` ID: `ghcr.io/ggml-org/llama.cpp@sha256:33868c035b21dc63f7c60b7438774283fd99215bc319114eb03de5df4ce7cd6b` ### Operating systems Linux ### GGML backends CPU ### Hardware AMD Ryzen 9 3900X 12-Core Processor ### Models -...
Eval bug: Vulkan + mmproj: ~250x slower generation with a UD-Q3_K_XL quant of Qwen3.5-35B-A3B (same projector is fine with the IQ3_XXS quant)
...--- *Reported from a machine where `Qwen3.5-35B-A3B-UD-Q3_K_XL` + `mmproj-F16` is unusable, while `UD-IQ3_XXS` + the same projector runs at 27.7 tok/s.*
Building domain agents: is the bottleneck getting SMEs’ time or getting them to agree?
Been talking to a few people building agents for finance and ops teams and the same thing keeps coming up. the eval / training data part isn’t really blocked by tooling, it’s blocked by getting the right answer out of the people who actually know the domain.. one guy told me...
How will LangChain continue making money?
...tools like AWS Omni and other observability platforms that solve the same problem, I’m failing to see the growth potential. Like, take a mid tier organization that already has a BAA with AWS. They can build all their agents on Agentcore, and connect all their various services....
Eval bug: Fail to run StartLux-Decision model, llama-server reports 501.
### Name and Version b11494 ### Operating systems Windows ### GGML backends CUDA ### Hardware Nvidia P620 ### Models _No response_ ### Problem description & steps to reproduce https://huggingface.co/startlux-models/StartLux-Decision-0.8B-Q8_0-GGUF...
Eval bug: llama-server crashes due to "bad allocation" during long conversations
### Name and Version >.\llama-server.exe --version version: 0.6.0-dev (build 11455, commit abeada33) built with Clang 24.0.0 for Windows ### Operating systems Windows ### GGML backends HIP ### Hardware Ryzen 9700X + 1x Radeon RX 9070 ### Models Qwen3.8-27B-IQ3_S-3.23bpw.gguf in...
[FR] Add aggregate metrics and summary statistics for MLflow Traces (e.g., total/average token count and cost per session or job)
> [!WARNING] > Before submitting a PR, please make sure that: > - A maintainer has triaged this issue and applied the `ready` label > - This issue has no assignee > - No duplicate PR exists > > PRs not meeting these requirements may be automatically closed. ## Proposal...
Wav2Vec2 and 6 other audio models leave masked_spec_embed uninitialized when the checkpoint lacks it
### System Info - `transformers` main (`4d536e1f`, still present on `e598fbad92`), torch 2.11.0, Python 3.10, macOS (Apple M5), CPU ### Reproduction `facebook/wav2vec2-base-960h` has no `wav2vec2.masked_spec_embed` tensor (SpecAugment's mask vector). On load, the missing...
/api/chat returns HTTP 500 after llama-server completes normally (unexpected end of JSON input) with qwen3.8:27b
...remains alive after the direct completion test Separately, the server log shows that the failing inference itself completed, released its slot, and returned to idle before Ollama reported the HTTP 500. This suggests that the failure occurs after model inference, somewhere in...
Eval bug: decision models fail on large input size
### Name and Version version: 0.6.0-dev (build 11457, commit 5ad1c5da0) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware 3090s GPU ### Models Clef Q8, Q4 and Kev Q8 ### Problem description & steps to reproduce When I run the...
Eval bug: llama.cpp server hangs (loops forever) at the same token when used from lm_eval
### Name and Version version: 0.6.0-dev (build 11461, commit 4d756bc72) built with Clang 20.1.8 for Windows x86_64 Tested with slightly older version as well (b11284) with the same result ### Operating systems Windows ### GGML backends CUDA ### Hardware Intel 4790K + Nvidia 1060...
Eval bug: --split-mode tensor output degenerates into repeated garbage tokens after 2-3 requests (meta backend, 2x Tesla P100)
### Name and Version version: 0.4.0 (commit 5266f24), built for sm_60, CUDA 12.4, driver 580 Tree: 44-patch sm_60 set (shinbunbun/llama-cpp-p100-patches) + qr0n 0003-0005 + #29266 + the on-demand registration fix from #29466 ### What happened? On 2× Tesla P100 (SM 60) with...
Eval bug: embeddinggemma-2 vision/audio encoders unreachable via server embeddings; /embedding silently ignores image_data
...= True`, projectors `gemma4v` / `gemma4a`), and it loads without complaint: But I can't find any way to get an image (or audio) embedding out of the server. Three request shapes, all with a 64x64 PNG: **1. `POST /v1/embeddings` with an OpenAI-style image part** → 400 **2. `POST...
Improve performance of per-instance `self` references with count/for_each
### Terraform Versions and Benchmark Results The aggregate `self` lookup path is present in Terraform v1.16.5 and v1.17.0-beta2: `internal/lang/eval.go` resolves `self` through `GetResource`, and `internal/terraform/evaluate.go` decodes all instances for that resource. I ran the...
Eval bug: Performance degradation since (PR #29622) with unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS and dual Intel B70
### Name and Version version: B11400, 0bb496d ### Operating systems Linux ### GGML backends SYCL ### Hardware VM CPU (32 Ryzen 9 5950X) 2x Intel B70 ### Models unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ3_XXS ### Problem description & steps to reproduce I'm getting a performance...
I feel the only people struggling with this are people who loved programming and syntax. Product-minded software engineers are thriving. Less tedium, easier refactors, manageable scope creep. The “hard” part of engineering has moved up to higher level challenges: evals,...
Explicit kernelization modes are ignored on evaluation models
### System Info - Transformers: current main `263c5eb913849847dcff1885b7b5e387c483f844`, `5.19.0.dev0`. - Python: `3.12.13`. - PyTorch: `2.11.0+cu129`. - kernels: both `0.17.0` and `0.17.2`, within the supported `>=0.17.0,<0.18.0` range. - Reproduced on Linux with CUDA devices...
[Bug][ROCm] Qwen3.8-27B-FP8 (TP4, MI355X) accuracy collapses at high concurrency with `VLLM_ROCM_USE_AITER=1`; fine with AITER off
### Your current environment The output of python collect_env.py ### 🐛 Describe the bug With AITER enabled, Qwen3.8-27B-FP8 at TP=4 on MI355X gives correct outputs at low concurrency. At ≥128 concurrent requests, outputs get much worse (GSM8K, greedy decoding): - GSM8K accuracy...
Eval bug:
### Name and Version version: 0.6.0-dev (build 11443, commit f0c41e016) built with GNU 13.3.0 for Linux aarch64 ### Operating systems Linux ### GGML backends CPU ### Hardware Google Axion, Neoverse V2 with SVE2, on GCP c4a-standard-16 (Intel Xeon Granite Rapids) ### Models Gemma...
Eval bug: Clef-GGUF Q8_0 /v1/systemone probabilities collapse toward uniform, while BF16 gives correct ones.
### Name and Version > llama --version version: 0.6.0 (build 11429, commit d81235049) built with AppleClang 21.0.0.21000334 for Darwin arm64 ### Operating systems Mac ### GGML backends Metal ### Hardware m5 Max Mac Studio w/128 GB of RAM ### Models Clef-GGUF & Kev-4B-GGUF @ Q8_0...
What benchmark do you wish someone would build, especially for multiagent system with MCP tools⁉️
...- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.. I’m especially interested in cases where you thought:. “My system desperately needs to do this in production, but I have no good way to benchmark it.”. What was...
...in every possible place not to do it and Codex instructed to catch these cases. It doesn’t work. Lots of sloppy evals get written and skills get extremely specific instructions to pass specific tests. I think if you are implementing trivial features in a green field product it...
What benchmark do you wish someone would build⁉️ such as traumatized by MAS with 40+ MCP tools
...- You knew what you wanted to test, but building a custom benchmark from scratch was too expensive or complicated.. I’m especially interested in cases where you thought:. “My system desperately needs to do this in production, but I have no good way to benchmark it.”. What was...
...not used in any user-facing decision.. The project does not need an audio channel that does not work. It already has a fusion engine that combines calibrated per-signal evidence across video, image, URL, and government document channels. Adding a component that scores 0.54 would...
...too. Standard loops are fine, and I'll also take a paid 2-3 week trial sprint on a real bottleneck in your codebase.
Eval bug: ggml-vulkan ggml_vk_wait_for_fence spins forever when the driver never reports device loss after a GPU fault
...instead of hanging. ### Repro note The trigger is a hardware or driver fault, so it is hard to reproduce on a healthy GPU. The defect is that the wait has no upper bound, whatever the trigger. 5 of 5 runs stalled on the tuned card and 0 of 5 at stock clocks, with gdb stacks...
MLX engine: weights are unwired ~2 s after each request on macOS 27, so the first request after idle pages them back in
...the GGUF engine keeps weights wired between requests and the MLX engine does not. ### Workaround `sudo sysctl iogpu.disable_wired_collector=1`. With it set, wired memory stayed at 37.3-37.4 GiB for three minutes after a request, where before it fell back to 2 GiB within seconds....
[ROCm]: run-to-run GSM8K accuracy variance for simple-nemotron-h-8b in KV-Offload test on MI300
### Name of failing test `tests/evals/gsm8k/test_gsm8k_offloading.py::test_gsm8k_offloading_correctness[simple-nemotron-h-8b]` ### Basic information - [x] Flaky test - [x] Can reproduce locally - [ ] Caused by external libraries (e.g. bug in `transformers`) ### 🧪 Describe the...
[RFC]: Handle empty responses in synthetic acceptance benchmarks
### Motivation. Agentic performance tests using synthetic speculative acceptance can generate tokens while delivering no usable response data to the benchmark client. Synthetic acceptance can accept draft tokens that normal verification would reject. The resulting output may...
Eval bug: GLM-5.3-Flash decode stalls on Metal because fused Lightning Indexer falls back to CPU
...the probe runs, and/or re-expose `LLAMA_FUSED_LID_DISABLE` as a manual escape hatch. **Workaround:** Build from source with the probe enabled in `src/llama-context.cpp`: [Line...
Eval bug: ggml-vulkan crash with vulkan 1.0
### Name and Version I'm using ggml (from master, circa v0.25.3) in my own software. ggml-vulcan requires Vulkan 1.2. To enforce this it calls (indirectly, via some templates) the vkEnumerateInstanceVersion function pointer from the Vulkan implementation and compares it to the...
Eval bug: Qwen3.8-27B + `--spec-type draft-mtp` + `--split-mode tensor` on ROCm (2x RX 9060 XT) hard-freezes the whole machine
### Name and Version version: 0.5.0-dev (build 11160, commit a3c12db9d) built with Clang 23.0.0 for Linux (Compiled by the Unsloth team) ### Operating systems Linux ### GGML backends HIP ### Hardware - CPU: AMD Ryzen 7 9700X (8 cores), 32 GB RAM - GPU: 2x AMD Radeon RX 9060 XT...
Eval bug: llama-bench - state_read_data: incompatible V transposition, failed to restore kv cache
### Name and Version version: 0.5.0-dev (build 11369, commit 4ebdf2c74) built with MSVC 19.50.35727.0 for x64 ### Operating systems Windows ### GGML backends Vulkan ### Hardware Intel B70 Arc Pro, Nvidia RTX 4060 TI ### Models...
Nightly evals: summary job reads green when no eval reports exist
### What happened? In `scripts/aggregate_evals.js` (lines 242-249 on main), when `findReports` returns nothing, the script logs "No reports found." and calls `process.exit(0)`. The comment just above it says "practically if current has no reports, something is wrong." The Run...
Eval bug: Assert at startup when running Qwen 3.8 flash with MTP
### Name and Version Commit: 42d958167a748f2c04b1f888e84e7a58f609ddcb Command line: llama server -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf --spec-draft-model mtp-Qwen3.8-Flash-Next-Q8_0.gguf -c 262144 --spec-type draft-mtp -lv 4 from...
Eval bug: llama-server recurrent state not cleared when it holds NaN, garbling future requests
...ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 ### Problem description & steps to reproduce A NaN fails to clear in a llama-server slot after a request, garbling future unrelated requests until restart. I believe this is caused by the current clear procedure doing ggml_scale_inplace(x, 0),...
Eval bug: Flash attention on CPU (one-chunk) overflows to inf/NaN, has F16 accumulator instead of F32.
### Name and Version ./llama-cli --version version: 0.5.0-dev (build 11235, commit 6c7a87f7e) built with GNU 13.3.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CPU ### Hardware i9-14900K ### Models Nemotron 3 Nano 4B Q8_0...
...output on your work against your evals. There is way too much subtlety in what does and doesn't work for a given problem, context/prompt, tool set and eval. I can tell you Fable is generally better than Haiku, but comparing similar tiers really does depend on your exact context.
Eval bug: [BUG] Unstable Tool Calling for Gemma 4 Models during Multi-line Streaming and Partial Parsing
..."dialect" (unquoted keys). This manifests in two ways: - Parsing Failures: The parser fails to correctly handle streaming tokens for complex JSON structures, leading to truncated strings or missing delimiters. - Execution Errors ("Invalid Diff"): Orchestrators/Harnesses receive...
Eval bug: ggml-cpu AMX mul_mat (since #19925) doesn't broadcast 2D weights over batched src1 and mis-scales byte offsets → garbage output with ≥2 parallel sequences and SIGSEGV in tileloadd (Qwen3.5/3.6, llama-server -np)
...out-of-bounds reads/writes, and SIGSEGV once the read passes the AMX buffer end. ### Workaround `--no-repack` (`-nr`), or keep one sequence per ubatch (`-np 1`). ### Possible fix Verified locally: the op-level repro passes, and Qwen3.6-35B-A3B / Qwen3.5-0.8B give identical...
Eval bug: GPUs choking on PCIe (VK device lost)
### Name and Version llama.cpp version: version: 0.5.0-dev (build 11223, commit 4da633776) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends Vulkan ### Hardware RTX 3090 + RX 7900 XT ### Models Qwen3.8 27B Q8_0 ### Problem description & steps...
qwen3.8: `think: "high"` / `"max"` silently run the default (`medium`); the model's top level `xhigh` is reachable only by a string outside the documented `think` values
...three return HTTP 200, and the output is byte-identical to `"medium"`. The client has no way to tell that its level was ignored. This contradicts the thinking docs, which say a string selects "a supported level from the model's `thinking.values`". Other code paths do reject...
...of curiosity. From my past experience, the latest I’ve seen were Opus-4.6 and GPT-5.5 struggle a lot with standard GHC Haskell, no extensions, nothing fancy. Not that they produced impeccable Python or Rust. But it appeared to take several turns for obvious expressions, while at...
Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some
...tcclaviger’s MXFP4 quant of Flash Next, which should make the QFN run around 2x faster.. Any suggestions for more indicative benchmarks? Anyone did a bigger comparison?. There are so many comments every week saying FP8 27B is better than low quants of QFN. I would consider...
Add --no-open option to crewai eval for headless use
### Feature Area Other: CLI evaluation workflow. ### Is your feature request related to an existing bug? NA. This is a usability extension to the recently introduced `crewai eval` command. ### Describe the solution you'd like Add `crewai eval --no-open` to print the evaluation...
[BUG] crewai eval sends saved organization ID to untrusted AMP origins
### Description `crewai eval` correctly avoids sending the saved bearer token when a project's `.env` points `CREWAI_PLUS_URL` at an untrusted origin. However, the replacement anonymous `PlusAPI()` client still loads `Settings().org_uuid` and adds it as...
Eval bug: VRAM usage increased from b10680 (SYCL backend)
### Name and Version Docker based llama.cpp (ghcr.io/ggml-org/llama.cpp:server-intel) : Build 11011 (aa39d7a3e) with IntelLLVM 2025.3.3 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL ### Hardware AMD Ryzen 9 5900XT + 2 x Intel Arc B580 ### Models...
Eval bug: SYCL demands extreme scratchpad allocation (2GB+) when ngram-mod is enabled
### Name and Version llama-server version: 0.4.0-dev (build 1, commit 4a89937) built with IntelLLVM 2026.1.1 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL ### Hardware Ryzen 5950X + Intel Arc A770 + AMD 6900XT ### Models Qwen3.8-27b from various sources and...
...solutions on the cheaper ones. edit: maybe not. Astra has imitated Fable, and appears unusable for biology due to safeguards.
...un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running...
everything server: instructions unconditionally reference sampling/elicitation tools that only exist when the client declares those capabilities
## Factual issue The `@modelcontextprotocol/server-everything` v2026.8.31 `instructions` string unconditionally tells integrating agents to use `trigger-sampling-request` and `trigger-elicitation-request`: These tools are registered through `registerConditionalTools()` inside...
...other post-training. They are highly sensitive to prompting, making sampling-based evals unreliable. Instead, we measured bits-per-byte loss on heldout data, which does not suffer from prompt sensitivity and smooths measurement of otherwise emergent abilities. As a side note, we...
Eval bug: DSpark + LFM2.5-8B-A1B (MoE) crashes at graph_reserve — dense LFM2.5 models unaffected
### Name and Version version: 0.4.0-dev (build 10853, commit 9dcf84e5a) ### Operating systems Linux ### GGML backends CPU ### Hardware ASUS Nuc 14 Pro. Intel Core Ultra Series 1 (155H) Meteor Lake. ### Models LFM2.5-8B-A1B-Q4_K_M.gguf LFM2.5-8B-A1B-DSpark-Q8_0.gguf ### Problem...
...article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these evals,...
Amazing work! But deeply frustrating on the lack of reproducibility and how far off this is from a proper SDLC. How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere? These evals...
EvaluationResult.passed asserts over non-verdict columns: dataset expectations always fail the run, and feedbacks logged without sourceRunId metadata by anyone are counted as this run's assertions
...expectation whose value happens to be a bool or yes/no string is counted as a passing or failing verdict, conflating ground truth with measurements. 2. Foreign feedback. The only provenance filter in `_get_assessment_values` is `if (source_run_id := a.get("metadata",...
Eval bug: K2-Horizon models fail to load
### Name and Version > llama-cli --version version: 0.3.0-dev (build 1, commit 95ef7fc) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware Intel(R) Core(TM) i9-14900HX NVIDIA GeForce RTX 4060 ### Models K2-Horizon-1B from...
...you mean a single minor incident involving a third party eval partner. I feel like you struggle to read. I wrote: "Intrusions by OpenAI models continued after the Hugging Face was published and acknowledged by OpenAI. They did not change their behaviour after multiple incidents,...
Eval bug: draft-mtp + --parallel > 1 causes cross-slot content contamination (not HIP-graphs related)
## Summary With `--spec-type draft-mtp` and `--parallel N` (N > 1), concurrent requests carrying clearly distinct prompts can produce responses that drift into content belonging to a *different* concurrent request's prompt/completion. Output stays valid UTF-8 with no...
Eval bug: Error on loading Gemma 4 family with the new updates
### Name and Version version: 0.3.0-dev (build 10744, commit 1f3d31873) built with GNU 15.2.0 for Linux x86_64 ### Operating systems Linux ### GGML backends CUDA ### Hardware 5070 ti+i9 14900k ### Models Gemma 4 family ### Problem description & steps to reproduce When I use MTP...
Eval bug: every mrope image decode on hybrid recurrent models (qwen3_5) triggers guaranteed non-consecutive position warnings in llama_memory_recurrent; mechanism fixed by #27941 for hybrid-idx only
### Name and Version Pinned fork at commit `d6d0ce821`. Before filing we checked the relevant code at current master (`9d817213a`): the `find_slot` warning block in `src/llama-memory-recurrent.cpp` and the memory-class selection for qwen3_5 in `src/llama-model.cpp` are...
...too. Standard loops are fine, and I'll also take a paid 2-3 week trial sprint on a real bottleneck in your codebase.
Eval bug: context type MTP requested but model doesn't contain MTP layers
### Name and Version daef7b6874397a5a7c3d7e38b55e2ee0adf7da38 ### Operating systems Linux ### GGML backends CUDA ### Hardware H100 ### Models https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/ ### Problem description & steps to reproduce context type MTP requested but model...
...and they have a ton of gaps about which models they’re doing their tests on because it’s so expensive. The idea behind any eval is so that you can pay the few pennies it costs to run an individual problem in an eval, and then as a together as a collective, we can crowdsource...
Claude Code for Research Papers [R]
...(For me I think the eval harness and anything defining a metric should stay mine, but I keep breaking my own rule.). . Does anyone have a workflow that keeps the speedup without the detachment? Reading the diff line by line is not cutting it.. . Not looking for a “tools are just...
Eval bug: llama-server crashes (stack overflow / access violation) on hybrid recurrent models (qwen3_5_moe) with mmproj vision when alternating text and image turns
### Name and Version Also reproduced on b10639 and on current master sources (checked `src/llama-memory-recurrent.cpp` and `tools/server/server-context.cpp` - identical in master). ### Operating systems Windows ### GGML backends CUDA, CPU ### Hardware Ryzen 9 9950X3D, 96 GB RAM,...
...to perform suspiciously well in public benchmarks vs private ones and what models still struggle with along with why, so yeah, certainly can always improve but I got, I'd argue, enough of an idea to where my critique of LLM coding limitations has legs. Which brings us to what I...
Eval bug: Dflash2: vision support is still broken
...testing in with qwen 3.8 in llama.cpp webui, I found out that vision support (with PDFs) is broken, while the native mtp is not. checkpoint from https://huggingface.co/incoai/Qwen3.8-27B-DFlash2-GGUF parameters (as suggested): model-draft = Qwen3.8-27B-DFlash2-Q4_K_M.gguf...
Eval bug: SYCL multi-GPU crash with Intel Arc Pro B50 + Arc A770
### Name and Version llama.cpp build: b10644-d7a207411 The executable was built locally from the ggml-org/llama.cpp repository. version: 0.3.0-dev (build 10644, commit d7a207411) built with IntelLLVM 2026.1.0 for Linux x86_64 ### Operating systems Linux ### GGML backends SYCL...
Eval bug: DFlash2: --split-mode tensor aborts with SPLIT_AXIS_UNKNOWN on ROCm (Qwen3.8-27B, 2x RX 7900 XTX)
### Name and Version Current master: b10660 6c84c7d5d model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) This includes the merged DFlash2 implementation from #27342. ### Operating systems Linux ### GGML backends HIP ### Hardware Ryzen 3600 Asus prime x570 pro (8x/8x) 2 x rx 7900...
Eval bug: crash on RPC server when using gemma-4-26B MTP head
### Name and Version (base) MacStudio1:~ alvise$ /opt/llama.cpp/llama-server --version version: 0.1.0-dev (build 10442, commit 9b0a2ce85) built with AppleClang 21.0.0.21000101 for Darwin arm64 ### Operating systems Mac ### GGML backends Metal ### Hardware 2x Mac Studio M4 Max...
[DOC] Clarify that Detect Issues analyzes trace data only, not source code
> [!WARNING] > Before submitting a PR, please make sure that: > - A maintainer has triaged this issue and applied the `ready` label > - This issue has no assignee > - No duplicate PR exists > > PRs not meeting these requirements may be automatically closed. ### Summary The...
Companies and products named
About this problem
Evidence
153 posts from 144 people in 30 places, about 14 a week over 73 days. Mostly on ggml-org/llama.cpp, Hacker News comments, r/MachineLearning. Tools named alongside: llama.cpp, Hugging Face, NVIDIA, GitHub.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit, Hacker News, Mastodon, Stack Overflow
Summary, model-written
PR reviewers lack evaluation results, while users running concurrent requests experience output drifting into content from other prompt requests.
How it was grouped
Posts that state a pain and match the "evals" rule. First post Aug 27, 2026, latest Nov 7, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
History