Problemsby IbisBad data slips through Apache Spark unnoticed
Data engineering2 posts from 2 people2d active+00 posts in the last 7 days, 1 the 7 days before (fading)Unverified
Posts per day
The posts behind it
2, newest firstParquet column index marks pages with values as null pages, so Spark drops rows### Checks - [x] I have checked that this issue has not already been reported. - [x] I have confirmed this bug exists on the [latest version](https://pypi.org/project/polars/) of Polars. ### Reproducible example ### Log output ### Issue description When running a `PySpark` query...pola-rs/polars · Hoeze1 reactions, 1 repliesOpen ↗Spark: copy-on-write `UPDATE ... WHERE <subquery>` can silently duplicate rows — race in `SparkCopyOnWriteScan.filter()` when both `Union` branches apply the runtime file filter concurrently (Spark 4.0.x / 4.1.x)### Apache Iceberg version 1.11.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 **Symptom.** A copy-on-write `UPDATE t ... WHERE EXISTS ( )` intermittently commits a snapshot that **adds a rewritten copy of every row in the table while deleting only the...apache/iceberg · ignaski1 repliesOpen ↗
Companies and products named
About this problem
Evidence
2 posts from 2 people in 2 places, about 1 a week over 35 days. Mostly on apache/iceberg, pola-rs/polars. Tools named alongside: Apache Iceberg, Polars, GitHub, PyPI.
Frustration Frustration 0 of 3· Seen on GitHub issues
How it was grouped
Posts that state a pain and match the "data-quality" rule about Apache Spark. First post Sep 7, 2026, latest Oct 11, 2026. Unverified: fewer than 3 posts, or one person or place only. Method
The same problem elsewhere
Other Apache Spark problems
History
steady to fading
Bad data slips through Apache Spark unnoticed (2 posts)