Problemsby GlyphBad data slips through Databricks unnoticed
3 posts from 3 people2d activeSteady, +1 this week
Posts per day
The posts behind it
We pointed an AI agent at 146M NYC taxi rows and asked it to build a Data Vault mart. Here's what broke.
...it generates and runs the jobs.. What the agent got right on its own:. - total_amount is broken in this dataset (≈0.3 in almost every row, 31 distinct values across 146M). The agent flagged it and proposed revenue = fare + extras + tax + tip + tolls.. - There is no natural trip...
I have made a complete end to end Azure databricks project
...All the reviews and suggestions are welcome, please advice me what more can I learn and how can I make it better.. In couple of days I will add small demonstration of API and streaming ingestion too, currently it is doing batch ingestion from ADLS GEN2.....
Anyone running actual SLOs on Lakeflow Jobs, or is it still “it failed, rerun it”?
...as everything else. The useful part wasn't the dashboard, it was that "this pipeline is flaky" became "this pipeline burned most of its budget in a week", which lands very differently with the people who own it.. What I can't tell is whether anyone does this seriously. Most of...
Companies and products named
About this problem
Evidence
3 posts from 3 people in 2 places, about 1 a week over 44 days. Mostly on r/databricks, r/dataengineering. Tools named alongside: Microsoft Azure, GitHub, Delta Lake.
Frustration Frustration 2 of 3· Seen on Reddit
How it was grouped
Posts that state a pain and match the "data-quality" rule about Databricks. First post Sep 22, 2026, latest Nov 4, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Bad data slips into production unnoticed
21 posts from 21 people on GitHub issues, Reddit, Stack Overflow.
Bad data slips through Polars unnoticed
3 posts from 3 people on GitHub issues, Reddit.
Bad data slips through Snowflake unnoticed
3 posts from 3 people on Reddit, Hacker News.
Bad data slips through Apache Iceberg unnoticed
3 posts from 3 people on GitHub issues.
Bad data slips through Apache Spark unnoticed
2 posts from 2 people on GitHub issues. Unverified.
Bad data slips through PostgreSQL unnoticed
2 posts from 2 people on Stack Overflow. Unverified.
Bad data slips through Claude unnoticed
2 posts from 1 person on Reddit. Unverified.
Other Databricks problems
Trouble with Databricks table
Top post: "Unity Catalog client re-encodes or decodes a table location that contains a percent sequence", one of 8 posts from 8 people on GitHub issues, Reddit.
Modeling data in Databricks is unclear and hard to change
7 posts from 7 people on Reddit.
Databricks jobs fail silently or break often
5 posts from 5 people on GitHub issues, Reddit.
Trouble with Databricks catalog
Top post: "Spark support for Unity Catalog Iceberg", one of 4 posts from 4 people on Reddit, GitHub issues.
Hard to trace lineage and find trusted data in Databricks
3 posts from 3 people on Reddit, GitHub issues.
Trouble with Databricks genie
Top post: "Databricks: support Genie consultations and resumable standalone tasks", one of 3 posts from 3 people on Reddit, GitHub issues.
Trouble with Databricks spark
Top post: "replaceUsing on partitioned tables fails with DELTA_FILE_TO_OVERWRITE_NOT_FOUND on local file:// paths", one of 3 posts from 3 people on GitHub issues.
Databricks tables suffer from small files and skew
3 posts from 3 people on Reddit, GitHub issues.
Databricks schema changes break downstream pipelines
3 posts from 3 people on GitHub issues, Reddit.
Databricks login, SSO or OAuth breaks
2 posts from 2 people on Stack Overflow, Reddit. Unverified.
Databricks vector search quality is hard to get right
2 posts from 2 people on Reddit. Unverified.
Databricks upgrades break working setups
2 posts from 2 people on GitHub issues. Unverified.
History