Skip to content
Problems.

Small files, skew and partitioning hurt performance

Data engineering9 posts from 9 people9d active+33 posts in the last 7 days, 0 the 7 days before (rising)

Posts per day

Posts per day9 posts, Aug 28 to Sep 25

The posts behind it

9, newest first
PostDate
Improve spark compaction scalability...performance for the table. When Flink job is under spikes, the Spark compactions are failing with OOM. **Root cause:** Compaction is using Spark iceberg reader, which loads equality deletes into memory in order to build a local filter. In hot partitions, there are huge amount of...apache/icebergqinghui-xuSep 24yesterday
How do you debug a slow Databricks query?...joins, partitioning, or cluster resources?. What steps do you follow to find the actual bottleneck?.r/databricksu/No_Ambition8323Sep 223 days ago
Image optimizer: a client abort permanently wedges that /_next/image cache key for the life of the process...and the final write to a dead socket fails the way any write to a dead socket does. Failing that, clearing the `Batcher` entry on a non-settling work function - or giving the internal sub-request a timeout - would at least stop one abort from being permanent. ### Workaround For...vercel/next.jsstrickddSep 205 days ago
UI: Ability to select model to be used for chat title summarization### Prerequisites - [x] I am running the latest code. Mention the version if possible as well. - [x] I carefully followed the [README.md](https://github.com/ggml-org/llama.cpp/blob/master/README.md). - [x] I searched using keywords relevant to my issue to make sure that I am...ggml-org/llama.cppNickM-27Sep 102 weeks ago
Support encryption in `rewrite_manifests`### Feature Request / Improvement Rewrite manifests writes plaintext manifests - this tracks supporting encryption for manifest compaction. ### Query engine None ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing...apache/icebergsmaheshwar-pltrSep 72 weeks ago
Perf: consolidate head/footer HTTP requests on Parquet, IPC### Description Every cloud Parquet/IPC metadata read costs **two** round trips: a `HEAD` to learn the file size, then a ranged `GET` for the metadata footer. The HEAD transfers zero bytes and exists only to compute the range offset. MinIO trace, `scan_parquet` over 5 files,...pola-rs/polarskdn36Sep 33 weeks ago
AI Assistant: mid-run observation compaction (#37234) sends two system messages, breaking strict OpenAI-compatible backends (System message must be at the beginning)...request:` prefix wraps the Jinja `raise_exception`). This makes the compaction feature unusable on a large class of self-hosted models, even though the compaction itself completes successfully. ### To Reproduce 1. n8n 2.38.1 (docker), AI Assistant with a custom OpenAI-compatible...n8n-io/n8nchrisqianzSep 23 weeks ago
Desktop app: bring back the context-usage indicator (or make the footer widget configurable)The desktop app's bottom-right footer currently shows the 5-hour usage-limit ring. Earlier builds showed context-window usage there. For long agentic sessions the context number is the more actionable one (am I approaching compaction?), while the usage ring rarely changes any...anthropics/claude-codeewokeeAug 303 weeks ago
Introduce model-based correctness and linearizability testing for k8s storage### What would you like to be added? We would like to introduce model-based correctness testing suite for Kubernetes storage (`k8s.io/apiserver/pkg/storage`), modeled after the [robustness testing approach used in...kubernetes/kubernetesserathiusAug 284 weeks ago

Companies and products named

Company or productPosts naming it
GitHub2
Apache Iceberg2
llama.cpp1
Claude Code1
Microsoft Azure1
Polars1
About this problem

Evidence

9 posts from 9 people in 8 places, about 2 a week over 28 days. Mostly on apache/iceberg, ggml-org/llama.cpp, anthropics/claude-code. Tools named alongside: GitHub, Apache Iceberg, llama.cpp, Claude Code.

Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit

How it was grouped

Posts that state a pain and match the "file-formats" rule. First post Aug 28, 2026, latest Sep 24, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

History

  • 2026-09-25 Corroboration: unverified to corroborated
  • 2026-09-25 Momentum: fading to rising
  • 2026-09-25 Added: Small files, skew and partitioning hurt performance (2 posts)

Rising problems by email

Mondays: the problems in data, tech and AI that grew fastest that week.

Double opt-in. Unsubscribe any time.