Data and AI struggles

1,730 problems · 342 companies

Data engineering

Small files, skew and partitioning hurt performance

14 posts from 14 people13d activeFading, +1 this week

Posts per day

Posts per day14 posts, Aug 28 to Nov 9

The posts behind it

GitHub issuesClickHouse/ClickHouse · fm4vNov 2

Iceberg Glue catalog: `INSERT` fails permanently with `Write into iceberg was not successful` after another writer (pyiceberg, Spark) commits, because ClickHouse names metadata files `v<N>.metadata.json` from the catalog pointer

### Describe what's wrong When a Glue Iceberg table is written both by ClickHouse (through a `DataLakeCatalog` database with `catalog_type = 'glue'`) and by any Iceberg Java or pyiceberg writer, ClickHouse writes stop working after a couple of alternating commits. Every further...

3 replies
GitHub issuesopen-webui/open-webui · zedr32Nov 1

feat: context compaction should reuse the prompt cache (summary request is always a full cold read)

...and Discussion #30105. ### Examples or References _No response_ ### Alternatives or Workarounds I run a small proxy between Open WebUI and the engine that spots the compaction request, rebuilds it as described above from the chat's last request, and falls back to the original if...

2 replies
GitHub issuesairbytehq/airbyte · da-saiyam-shahOct 28

[source-sftp-bulk] EMPTY_STREAM for root folder "/" with a relative glob (duplicate-slash path bug)

...discovers no files and fails with EMPTY_STREAM, even though a matching file exists. Failing configuration: { "folder_path": "/", "streams": [ { "name": "Lacey", "globs": ["downloaded/client2/Lacey*"] } ], "start_date": "2026-10-01T00:00:00.000000Z" } Target file:...

GitHub issueshashicorp/terraform · rickeyskiOct 21

Offer option to Compress plan output as zstd instead of deflate

### Terraform Version ### Use Cases We currently run almost 1000 root modules in our environment. Our average plan file size is 30mb. This adds up to a lot of bandwidth used when we transfer the plans from our CI runners in our cloud environment to our source control. There is...

3 reactions1 replies
GitHub issuesopen-webui/open-webui · Classic298Oct 21

feat: hide toggles and buttons of disabled features, and let admins hide UI elements

**Problem** Open WebUI shows many toggles, buttons and menu entries for features that are turned off for the whole instance. Clicking them does nothing, opens an empty list or shows an error. Admins also cannot hide elements their users do not need without turning off the...

2 reactions
GitHub issuesduckdb/duckdb · angrydengOct 10

Segfault in PhysicalBatchCopyToFile on VARCHAR data during COPY ... ORDER BY TO parquet (1.5.2 and 1.5.6)

### What happens? `COPY (SELECT ... ORDER BY ...) TO '...parquet'` intermittently segfaults inside `PhysicalBatchCopyToFile` while handling `VARCHAR` data. The pattern looks like a dangling `string_t`. It reproduces on 1.5.2 and on 1.5.6, but the crash point differs between the...

GitHub issueschroma-core/chroma · gpulipatiOct 6

[Bug]: delete_collection() leaves orphaned HNSW segment directory after successful deletion

### What happened? We are running ChromaDB in persistent HTTP/server mode and observed that `delete_collection()` successfully removes the collection from Chroma's logical catalog, but the corresponding persisted HNSW vector segment directory remains on disk. This was reproduced...

GitHub issuespola-rs/polars · laughingman7743Oct 4

read_csv(use_pyarrow=True) fails or returns wrong values on quoted values with newlines

...passes user options to `pl.read_csv()` (pyathena-dev/PyAthena#1048). pandas has the same issue with `engine="pyarrow"` (pandas-dev/pandas#56396, pandas-dev/pandas#59009). ### Expected behavior `read_csv(..., use_pyarrow=True)` reads quoted values that contain newlines, as the...

GitHub issuesollama/ollama · ChronoVarienceOct 3

Windows installer-signature bug: Ollama v0.35.1 Windows installer fail Authenticode with HashMismatch

...would defeat the purpose of the installer integrity validation and is not an appropriate workaround. ## Reproduction summary 1. On Windows, download `v0.35.1/OllamaSetup.exe`. 2. Run: 3. Observe: 4. Repeat with `v0.35.0/OllamaSetup.exe`. 5. Observe: This appears specific to the...

GitHub issuesmistralai/mistral-vibe · mdeweerdSep 30

Compaction fails with 'Session lifecycle transition is active' error

When attempting to compact a session near the token limit, the operation fails with an error indicating a session lifecycle transition is already active. **Steps to Reproduce:** 1. Fill a session to near token limit (199k/200k tokens) 2. Execute `/compact` command 3. Observe...

GitHub issuesggml-org/llama.cpp · NickM-27Sep 10

UI: Ability to select model to be used for chat title summarization

### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...

2 reactions
GitHub issuespola-rs/polars · kdn36Sep 3

Perf: consolidate head/footer HTTP requests on Parquet, IPC

### Description Every cloud Parquet/IPC metadata read costs **two** round trips: a `HEAD` to learn the file size, then a ranged `GET` for the metadata footer. The HEAD transfers zero bytes and exists only to compute the range offset. MinIO trace, `scan_parquet` over 5 files,...

1 reactions
GitHub issueskubernetes/kubernetes · serathiusAug 28

Introduce model-based correctness and linearizability testing for k8s storage

### What would you like to be added? We would like to introduce model-based correctness testing suite for Kubernetes storage (`k8s.io/apiserver/pkg/storage`), modeled after the [robustness testing approach used in...

9 replies

Companies and products named

5 posts
2 posts
1 post
1 post
About this problem

Evidence

14 posts from 14 people in 12 places, about 1 a week over 70 days. Mostly on pola-rs/polars, open-webui/open-webui, ggml-org/llama.cpp. Tools named alongside: GitHub, Polars, llama.cpp, Microsoft Azure.

Frustration Frustration 0 of 3· Seen on GitHub issues, Hacker News

How it was grouped

Posts that state a pain and match the "file-formats" rule. First post Aug 28, 2026, latest Nov 5, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

The same problem elsewhere

Claude Code tables suffer from small files and skew

9 posts from 9 people on GitHub issues, Reddit.

ChatGPT tables suffer from small files and skew

5 posts from 5 people on GitHub issues.

GitHub tables suffer from small files and skew

5 posts from 5 people on GitHub issues.

Claude tables suffer from small files and skew

4 posts from 4 people on GitHub issues, Hacker News.

Codex tables suffer from small files and skew

4 posts from 4 people on GitHub issues.

OpenAI tables suffer from small files and skew

3 posts from 3 people on GitHub issues.

Databricks tables suffer from small files and skew

3 posts from 3 people on Reddit, GitHub issues.

Microsoft Fabric tables suffer from small files and skew

2 posts from 2 people on GitHub issues, Stack Overflow. Unverified.

Apache Flink tables suffer from small files and skew

2 posts from 2 people on GitHub issues. Unverified.

Apache Iceberg tables suffer from small files and skew

2 posts from 2 people on GitHub issues. Unverified.

Vercel tables suffer from small files and skew

2 posts from 2 people on GitHub issues. Unverified.

History

Momentum went from steady to fading.

Nov 9, 2026

Momentum went from rising to steady.

Nov 8, 2026

Momentum went from steady to rising.

Nov 6, 2026

Momentum went from fading to steady.

Nov 5, 2026

Momentum went from steady to fading.

Oct 30, 2026

Momentum went from rising to steady.

Oct 28, 2026

Momentum went from steady to rising.

Oct 27, 2026

Momentum went from fading to steady.

Oct 25, 2026

Momentum went from steady to fading.

Oct 22, 2026

Momentum went from fading to steady.

Oct 18, 2026

Momentum went from steady to fading.

Oct 17, 2026

Momentum went from fading to steady.

Oct 16, 2026