Skip to content
Problems.

Bad data slips into production unnoticed

Data engineering7 posts from 7 people7d active+33 posts in the last 7 days, 0 the 7 days before (rising)

Posts per day

Posts per day7 posts, Sep 1 to Sep 25

The posts behind it

7, newest first
PostDate
Replace duplicated DataFrame indices with a single Series...dfdup.where(mask, "").max() In: dfresults Out: Sample1 Yellow Sample2 Yellow Sample3 Red How do I replace the 2 rows of df["A"] with the series dfresults without dropping duplicate rows (dropping duplicates from all of df would also affect "B" which hasn't been processed yet).....Stack Overflow [pandas]Kes PerronSep 223 days ago
[Spec][Parquet] Remove "Omit from data files" requirement for `UNKNOWN` type...manifests: - STRUCT (this includes the schema root) with all fields omitted: there is no way to indicate the distinction between a row being `NULL` or `{}`. - LIST with omitted element: there is no way to indicate whether a list is `NULL` or `[]`. This has (presumably...apache/icebergTishjSep 205 days ago
CustomWebhookNotificationBlock rejects JSON null values during initialization### Bug summary `CustomWebhookNotificationBlock` cannot be constructed when `json_data` contains a JSON null (`None`), including nested object fields or list items. It raises `ValueError: Unexpected type: ` before any notification request is sent I reproduced this on the latest...PrefectHQ/prefectHaokaiDingSep 196 days ago
I Built a Data Quality Library Because Existing Tools Were Too Slow for My PipelinesI got mass fed up with data quality tools, so I built my own I work as a data engineer....DEV #dataengineeringvladislav_koval_f4bed0e76Sep 112 weeks ago
Looking for advice on an unusual workflowBackground: . My team does a lot of short-term data analysis projects. We have a Snowflake db with a lot of 'base' tables from other departments or organisations, pretty much all poorly documented and with questionable data quality. The underlying data often changes for months...r/dataengineeringu/Squeaky_SqueakinsonSep 82 weeks ago
Spark: copy-on-write `UPDATE ... WHERE <subquery>` can silently duplicate rows — race in `SparkCopyOnWriteScan.filter()` when both `Union` branches apply the runtime file filter concurrently (Spark 4.0.x / 4.1.x)### Apache Iceberg version 1.11.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 **Symptom.** A copy-on-write `UPDATE t ... WHERE EXISTS ( )` intermittently commits a snapshot that **adds a rewritten copy of every row in the table while deleting only the...apache/icebergignaskiSep 72 weeks ago
find_orm_dags: multiple joinedload() collections cause row-explosion, slow DagModel sync (commonly >10s)...fast enough. That points at result-set size, not query planning/execution cost, as the bottleneck. This customer reports these commonly exceeding 10 seconds. Response latency downstream (this call sits in the request path of the DAG-processor's parse-result heartbeat) has been...apache/airflowseanmuthSep 13 weeks ago

Companies and products named

Company or productPosts naming it
Apache Iceberg2
Snowflake1
Excel1
pandas1
Apache Airflow1
PostgreSQL1
About this problem

Evidence

7 posts from 7 people in 6 places, about 2 a week over 22 days. Mostly on apache/iceberg, DEV #dataengineering, r/dataengineering. Tools named alongside: Apache Iceberg, Snowflake, Excel, pandas.

Frustration Frustration 2 of 3· Seen on GitHub issues, DEV, Reddit, Stack Overflow

How it was grouped

Posts that state a pain and match the "data-quality" rule. First post Sep 1, 2026, latest Sep 22, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

History

  • 2026-09-25 Added: Bad data slips into production unnoticed (6 posts)

Rising problems by email

Mondays: the problems in data, tech and AI that grew fastest that week.

Double opt-in. Unsubscribe any time.