Problemsby IbisDatabricks tables suffer from small files and skew
Data engineering3 posts from 3 people3d active+22 posts in the last 7 days, 0 the 7 days before (rising)
Posts per day
The posts behind it
3, newest first[Docs] Document `spark.databricks.delta.optimize.maxThreads` in Compaction (bin-packing) sectionPage: https://docs.delta.io/optimizations-oss/#compaction-bin-packing Description The note block in the Compaction (bin-packing) section documents spark.databricks.delta.optimize.repartition.enabled. It does not mention spark.databricks.delta.optimize.maxThreads, which sets the...delta-io/delta · kckohOpen ↗Small high frequent data ingestion cheaply...data to be available in our data products near realtime.. Databricks makes it extremely hard to achieve this goal.. I tried doing 5 min schedule poc pipeline and the bill was 1000$ for 1 pipeline per month which is insane.. In this poc i did streaming table raw(reads from...r/databricks · u/PerceptionBig940Open ↗How do you debug a slow Databricks query?...joins, partitioning, or cluster resources?. What steps do you follow to find the actual bottleneck?.r/databricks · u/No_Ambition8323Open ↗
Companies and products named
About this problem
Evidence
3 posts from 3 people in 2 places, about 1 a week over 31 days. Mostly on r/databricks, delta-io/delta. Tools named alongside: Apache Spark, Microsoft Azure, Delta Lake.
Frustration Frustration 2 of 3· Seen on Reddit, GitHub issues
How it was grouped
Posts that state a pain and match the "file-formats" rule about Databricks. First post Sep 22, 2026, latest Oct 22, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Small files, skew and partitioning hurt performanceClaude Code tables suffer from small files and skewOpenAI tables suffer from small files and skewChatGPT tables suffer from small files and skewModel Context Protocol tables suffer from small files and skewApache Iceberg tables suffer from small files and skewApache Flink tables suffer from small files and skewGitHub tables suffer from small files and skewVercel tables suffer from small files and skewCodex tables suffer from small files and skewClaude tables suffer from small files and skew
Other Databricks problems
Databricks users keep asking for helpTrouble with Databricks sparkTrouble with Databricks tableModeling data in Databricks is unclear and hard to changeDatabricks jobs fail silently or break oftenTrouble with Databricks incrementalDatabricks vulnerabilities are hard to triage and patchBad data slips through Databricks unnoticedDatabricks login, SSO or OAuth breaksDatabricks vector search quality is hard to get rightDatabricks upgrades break working setupsHard to trace lineage and find trusted data in Databricks
History
Databricks tables suffer from small files and skew (3 posts)