Data and AI struggles

1,787 problems · 347 companies

Outages and change

ClickHouse upgrades break working setups

31 posts from 20 people19d activeFading, +0 this week

Posts per day

Posts per day31 posts, Sep 21 to Nov 11

The posts behind it

GitHub issuesClickHouse/ClickHouse · snelheidaiNov 4

Iceberg mutations write position delete files that reads apply to many data files; one file per data file cuts `apply_patch_parts_merge` reads by 89–91%

### Describe the situation Iceberg `ALTER TABLE ... UPDATE/DELETE` writes one position delete file per partition with the deleted rows of every data file it touches, so later reads apply each such file to many data files and read it for each. Writing one delete file per data...

1 reactions1 replies
GitHub issuesClickHouse/ClickHouse · groeneaiNov 4

DEFAULT of a column added by ALTER is evaluated in the global context for rows written before the ALTER (MergeTree, Memory)

### Describe the issue For rows written before `ALTER TABLE ... ADD COLUMN c ... DEFAULT `, the default is evaluated at read time in a copy of the global context, not in the context of the reading query. MergeTree does this for a part without the column...

GitHub issuesClickHouse/ClickHouse · cardy31Nov 3

`_server_credentials_allowed` changes the identity of a `disk(...)` declared before 26.7, so it can't be re-declared after an upgrade

...rejects. There seems to be no SQL workaround: - The injector strips any marker the user writes before it decides whether to add one ([DiskFromAST.cpp#L214](https://github.com/ClickHouse/ClickHouse/blob/v26.8.34.13-lts/src/Disks/DiskFromAST.cpp#L214)). - Without the session...

GitHub issuesClickHouse/ClickHouse · filimonovNov 3

Failed ALTER RENAME COLUMN rollback on MergeTree leaves the rename mutation applied against mismatched metadata

Two holes in the rollback of a failed `ALTER … RENAME COLUMN` on plain `MergeTree` (`StorageMergeTree::alter`, the `catch` after `addPreparedMutationEntry` / `setProperties`). Both leave the table in a state where a merge applies the rename mutation against metadata that does...

GitHub issuesClickHouse/ClickHouse · bugginsNov 3

A `CREATE` that fails to publish the table leaves the inner tables of a `MaterializedView` or `TimeSeries` table

### Company or project name _No response_ ### Describe what's wrong A `CREATE` of a `MaterializedView` with an inner table, or of a `TimeSeries` table, that fails while the table is being published leaves the inner tables it has already created in the database, without an owner....

GitHub issuesClickHouse/ClickHouse · mtomaNov 3

DNSCacheUpdater spins forever (100% CPU, holds DNSResolver mutex) in `CaresPTRResolver::wait_and_process` after the DNS server port becomes unreachable; `SYSTEM DROP DNS CACHE` and shutdown hang

### Describe what's wrong When the background `DNSCacheUpdater` refreshes a cached **reverse** (PTR) entry and the DNS server's UDP port becomes closed (the server answers with ICMP port unreachable, e.g. while the DNS service restarts), the updater thread never leaves...

1 replies
GitHub issuesClickHouse/ClickHouse · filimonovNov 3

Growing a large PODArray is 1.4–2× slower and needs ~1.5× more memory since mmap/mremap left Allocator (#52792)

**Company or project name** **Describe the situation** Since 23.9 (#52792, first attempt #51686 reverted in #52589) `Allocator::realloc` goes through jemalloc for every size. A large buffer that grows by doubling (`PODArray::push_back` / `insert`, `HashTable::resize`,...

1 replies
GitHub issuesClickHouse/ClickHouse · fm4vNov 2

Wide parts with a `Geometry` column written before 26.8 are unreadable after the upgrade: `Stream g.MultiPoint.size0 ... is not found`

...the part and other variant subcolumns (`g.Point`, `g.LineString`) works. There is no SQL workaround on 26.8: `ALTER ... MODIFY COLUMN` and `OPTIMIZE` need to read the column. Rolling back to 26.6 makes the data readable again, but a part merged on 26.6 again has no `MultiPoint`...

3 replies
GitHub issuesClickHouse/ClickHouse · snelheidaiOct 27

Answer repeated count() with a filter from exact per-granule counts in the query condition cache

### Describe the situation A repeated `SELECT count() FROM t WHERE ` still reads every granule that has a match, because the query condition cache keeps one bit per granule. Keeping the exact number of matching rows per granule lets `count()` add them up instead of reading....

2 replies
GitHub issuesapache/superset · bmuraykhiOct 27

ClickHouse: Drill to detail and SQL Lab table preview fail on AggregateFunction columns

### Bug description ### Bug description On ClickHouse tables with `AggregateFunction(...)` columns (the usual AggregatingMergeTree rollup), Drill to detail, dataset samples and the SQL Lab table preview fail with: These features select every column as-is, so the query returns...

GitHub issuesClickHouse/ClickHouse · seva-potapovOct 26

Iceberg INSERT doesn't cut data files at `iceberg_insert_max_bytes_in_data_file` when Parquet encodes in parallel (regression from #127670)

Since #127670, `MultipleFileWriter` starts a new data file when `buffer->count()`, the number of bytes already written to the current file, reaches `max_data_file_num_bytes`...

1 replies
GitHub issuesClickHouse/ClickHouse · jsc0218Oct 22

ppc64le: bundled libunwind unw_getcontext leaves callee-saved vector registers byte-swapped (fixed in LLVM main, not backported)

On little-endian ppc64le, `unw_getcontext` in contrib/llvm-project/libunwind/src/UnwindRegistersSave.S corrupts the caller's callee-saved vector registers. PPC64_STVS(n) does `xxswapd n,n; stxvd2x n,0,4` and never swaps back, so v20-v31 and f14-f31 come back byte-swapped. Fixed...

3 replies
GitHub issuesClickHouse/ClickHouse · vitlibarOct 16

RFC: Compression codecs for subcolumns

...`ZSTD` for a string); today one `CODEC` clause applies to every leaf, and the only workaround is to split the column. ### Syntax In `CREATE TABLE`, a keyword-first entry: the keyword `CODEC`, the name of the target, and the codec chain in parentheses. Like `INDEX`, `CONSTRAINT`,...

1 reactions7 replies
GitHub issuesClickHouse/ClickHouse · clickgapaiOct 15

`url` `ArrowStream` reads fail on HEAD 501: new seekability probe rethrows

...= 0` avoids it. - Any server with chunked/close-delimited GET and a failing HEAD breaks every `url`/`URL`-engine `ArrowStream` query. Root cause: `ArrowIPCBlockInputFormat::readStreamBatch` now calls `MessageReader::readBody` which unconditionally calls `checkIfActuallySeekable`...

GitHub issuesClickHouse/ClickHouse · clickgapaiOct 15

`comparisonCanThrow` skips `Variant` inside `Tuple`: reordered `WHERE` throws

...values holding `Variant`/`Dynamic` (for example behind a JSON/string predicate) starts failing with `NO_COMMON_TYPE` after the upgrade; workaround `short_circuit_function_evaluation_reorder_arguments = 0`. The same misclassification reaches the `force_enable` lazy-argument...

GitHub issuesClickHouse/ClickHouse · clickgapaiOct 15

Heavy argument shared by several `and` is now evaluated per `and`: 2.9x slower

### Describe what's wrong - A heavy predicate shared as the first argument of several `and`/`or` calls runs once per call instead of once overall. - Triggered when `short_circuit_function_evaluation_reorder_arguments` is 1 (the default) and several `countIf` calls share the same...

2 replies
GitHub issuesClickHouse/ClickHouse · clickgapaiOct 14

Accepted `disk(...)` definition rejected on RESTORE and re-ATTACH

...and (by code, not run) a new `Replicated` database replica cannot create it. Loud error; workaround is ` 1 ` in the server config, which also disables the server-settings check. ### Does it reproduce on most recent release? No - only `master` (`a5b019f44f7b`). The bug starts...

GitHub issuesBerriAI/litellm · duhdOct 14

Lens worker claims job but never progresses, fails after ~15 min with "Worker disconnected repeatedly"

...surface a clear, actionable error instead of silently retrying for 15 minutes and then failing with a generic message that gives no indication of *why* the worker disconnected. **Verified**: the `ValidationError` warning in the worker log; every run failing with `"Worker...

GitHub issuesClickHouse/ClickHouse · clickgapaiOct 13

`additional_result_filter` runs over the extremes stream: `LOGICAL_ERROR`

### Describe what's wrong - With `extremes = 1`, `additional_result_filter` evaluates the predicate on the synthetic min/max extremes rows as if they were data. - Trigger: any `SELECT` with `extremes = 1` and `additional_result_filter`, for example filter `a + b >= 2` over...

2 replies
GitHub issuesClickHouse/ClickHouse · AlgunenanoOct 7

Update the zstd fork and evaluate the pending upstream decompression work

### Why `ZSTD(3)` is now the default compression codec of MergeTree columns, marks and primary keys, so zstd decompression is the largest part of many scans. For example, in `SELECT COUNT(DISTINCT SearchPhrase) FROM hits` (ClickBench Q5) with the radix partitioned `uniqExact` of...

1 reactions
GitHub issuesClickHouse/ClickHouse · PedroTadimOct 6

Parquet v3 reader: a negative `num_values` in a column-chunk footer trips `chassert(diff.by_stage[i] >= 0)` in `ReadManager`

🕵️ Reading a Parquet file whose footer declares a negative `ColumnMetaData.num_values` for a column chunk aborts `clickhouse` on debug and sanitizer builds. The abort is a `chassert` in the v3 reader's memory accounting: reached from `ReadManager::flushMemoryUsageDiff`...

3 replies
GitHub issuesClickHouse/ClickHouse · zlareb1Oct 5

Background column-TTL merge wedges forever when a dependent MATERIALIZED column throws on the reset default (regression from #117815)

A background column-TTL merge permanently wedges when a dependent `MATERIALIZED` column's expression throws on the source column's default value (e.g. `intDiv(100, x)` when the TTL resets `x` to `0`). Introduced on master by #117815: the recompute added there runs inside the TTL...

GitHub issuesClickHouse/ClickHouse · zlareb1Oct 5

Pending async inserts into `Distributed` become permanently undeliverable after `RENAME COLUMN` and wedge all later sends

**TL;DR:** With default settings, an `INSERT` into a `Distributed` table is acknowledged and queued for background delivery. A later, perfectly legal `ALTER TABLE ... RENAME COLUMN` (applied to the local and the `Distributed` table) makes every pending file permanently...

GitHub issuesClickHouse/ClickHouse · clickgapaiOct 3

Compact codec proof reads only the first frame: `NONE`-default part refused

...because the first-frame check in the Compact leg fails to detect the default codec. Actual output of the reproducer above on `master` (`4127c65654d0`): Suggested fix In the Compact leg, walk the frames of `data.bin` with `getCompressionCodecForFile(..., skip_to_next_block =...

GitHub issuesClickHouse/ClickHouse · snelheidaiSep 25

hasAny over Array(String) compares every element with every constant needle: a needle hash set cuts a 32-needle hasAny from 5.4 s to 0.52 s

### Describe the situation `hasAny(arr, [constants])` over an `Array(String)` column compares every element with every needle through a virtual `IColumn::compareAt`, because `sliceHasImplAnyAll` is specialised only for numeric arrays. A row without a match therefore costs...

1 reactions
GitHub issuesBerriAI/litellm · tombiiSep 24

[Bug]: langfuse_otel builds OTLP endpoint without required /v1/traces suffix — self-hosted Langfuse traces silently dropped (404)

...`/api/public/otel` suffix (as opposed to the generic `/v1/traces` default) - there is no way to fix this via env vars today for `langfuse_otel` short of overriding the entire endpoint including the correct suffix, which the current code does not support (the `/api/public/otel`...

GitHub issuesClickHouse/ClickHouse · clickgapaiSep 23

Text index prunes matching rows: decline keys on needle padding, not value's

### Describe what's wrong - `WHERE s = toFixedString(...)` on a `String` column silently prunes rows whose value equals the needle plus trailing zero bytes. - Triggered by `text(tokenizer = array)` or `text(tokenizer = splitByString([sep]))` indexes when `use_skip_indexes = 1`,...

GitHub issuesClickHouse/ClickHouse · patelsmit32123Sep 22

Reuse checksum-matching detached parts when bootstrapping a fresh replica

### Use case When creating a fresh ReplicatedMergeTree replica, we want to seed it from an object-storage backup instead of transferring the entire dataset from peers. Backup parts can be restored into the table’s detached/ directory. Attaching them normally may allocate new...

1 reactions3 replies
GitHub issuesClickHouse/ClickHouse · qoegaSep 22

BACKUP/RESTORE ... TABLE db.t AS t loses the rename through parseQueryToJSON / formatQueryFromJSON (RESTORE target silently changes database)

`BACKUP`/`RESTORE ... TABLE db.t AS t` loses the rename through `parseQueryToJSON` / `formatQueryFromJSON`. For `RESTORE` this changes what the query does: the `AS` clause is the destination, so `RESTORE TABLE d.t AS t FROM ...` (restore into the current database) silently...

GitHub issuesClickHouse/ClickHouse · clickgapaiSep 22

ALTER NAMED COLLECTION DELETE lacks dependency check, breaks table reload

...engine definition needs ` ` from `nc`. - The table keeps working after the ALTER but fails to load on next ATTACH or server restart with a required-keys error. - `DROP NAMED COLLECTION` correctly refuses this same case with NAMED_COLLECTION_IS_USED, but ALTER DELETE does not. -...

GitHub issuesClickHouse/ClickHouse · alexey-milovidovSep 21

Implement compact snapshot files for `plain_rewritable` disks to speed up initialization and updates.

### Company or project name ClickHouse ### Describe the situation Initialization of a `plain_rewritable` disks can take a lot of time, as it has to list the metadata directory, read all `prefix.path` files, and list all the related directories. It can be tens of thousands of S3...

Companies and products named

31 posts
20 posts
1 post
1 post
About this problem

Evidence

31 posts from 20 people in 3 places, about 4 a week over 45 days. Mostly on ClickHouse/ClickHouse, BerriAI/litellm, apache/superset. Tools named alongside: GitHub, Text, Langfuse, Kubernetes.

Frustration Frustration 0 of 3· Seen on GitHub issues

Summary, model-written

ClickHouse users face performance issues and unexpected behavior with array comparisons, default ZSTD compression, table codecs, and Iceberg delete files.

How it was grouped

Posts that state a pain and match the "breaking-changes" rule about ClickHouse. First post Sep 21, 2026, latest Nov 4, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

The same problem elsewhere

GitHub upgrades break working setups

89 posts from 77 people on GitHub issues, Hacker News, Stack Overflow.

Upgrades and new releases break working setups

77 posts from 63 people on GitHub issues, Reddit, App Store reviews.

Claude Code upgrades break working setups

68 posts from 61 people on GitHub issues, Hacker News, Reddit.

Codex upgrades break working setups

67 posts from 60 people on GitHub issues, Reddit.

DuckDB upgrades break working setups

45 posts from 35 people on GitHub issues.

Claude upgrades break working setups

35 posts from 34 people on GitHub issues, Reddit, Hacker News.

Anthropic upgrades break working setups

27 posts from 22 people on GitHub issues.

PyTorch upgrades break working setups

26 posts from 21 people on GitHub issues, Reddit.

ChatGPT upgrades break working setups

26 posts from 26 people on GitHub issues, Community forums, App Store reviews.

OpenAI upgrades break working setups

25 posts from 25 people on GitHub issues, Reddit, Community forums.

Open WebUI upgrades break working setups

25 posts from 2 people on GitHub issues.

Gemini upgrades break working setups

24 posts from 24 people on GitHub issues, Community forums, Reddit.

Other ClickHouse problems

ClickHouse users keep filing bugs

Top post: "Tighten seccomp policy for optional kernel interfaces", one of 19 posts from 8 people on GitHub issues.

Trouble with ClickHouse column

Top post: "MaterializedPostgreSQL: reading a Nullable column's .null subcolumn (x IS NULL, count(x)) fails with", one of 15 posts from 9 people on GitHub issues.

ClickHouse costs outgrow the value it gives

11 posts from 3 people on GitHub issues.

Trouble with ClickHouse table

Top post: "`SYSTEM RESTORE REPLICA` on a second replica over-bumps the table `metadata` znode version; every later", one of 8 posts from 5 people on GitHub issues.

Experiments on ClickHouse are hard to reproduce and track

8 posts from 1 person on GitHub issues. Unverified.

Trouble with ClickHouse tables

Top post: "Paimon: resolve columns by field id, so tables with renamed or re-added columns are readable", one of 7 posts from 6 people on GitHub issues.

Trouble with ClickHouse join

Top post: "Explicit grace_hash now spills at max_bytes_ratio_before_external_join = 0.5 although the build side fits in", one of 5 posts from 4 people on GitHub issues.

ClickHouse error messages are cryptic

5 posts from 3 people on GitHub issues.

Trouble with ClickHouse read

Top post: "ORC: a corrupt PostScript `metadata_length` makes ORC predicate pushdown read `stripe_stats` out of bounds,", one of 4 posts from 4 people on GitHub issues.

Trouble with ClickHouse database

Top post: "`DROP NAMED COLLECTION` succeeds while a database uses the collection, and the server then fails to start", one of 4 posts from 2 people on GitHub issues.

Trouble with ClickHouse query

Top post: "Memory tracker counts jemalloc's freed pages as used, so a heavy query started right after another spills to", one of 4 posts from 3 people on GitHub issues.

ClickHouse rate limits and usage caps block work

3 posts from 3 people on GitHub issues.

History

Momentum went from rising to fading.

Nov 10, 2026

Summary written by a model: ClickHouse users face performance issues and unexpected behavior with array comparisons, default ZSTD compression, table codecs, and Iceberg delete files.

Nov 7, 2026

Momentum went from fading to rising.

Nov 6, 2026

Momentum went from steady to fading.

Nov 3, 2026

Momentum went from rising to steady.

Nov 2, 2026

Momentum went from steady to rising.

Oct 30, 2026

Momentum went from fading to steady.

Oct 28, 2026

Momentum went from steady to fading.

Oct 22, 2026

Momentum went from rising to steady.

Oct 21, 2026

Momentum went from steady to rising.

Oct 19, 2026

Momentum went from fading to steady.

Oct 18, 2026

Momentum went from steady to fading.

Oct 13, 2026