Problemsby GlyphBad data slips into production unnoticed
21 posts from 21 people20d activeSteady, +1 this week
Posts per day
The posts behind it
[Bug]: [REST, 1.5.9] Empty metadata arrays are silently dropped in boundary-position and all-empty batches
### What happened? The REST API returns 201 when a batch contains `items: []`. Reading the records back returns no `items` key for each empty-array row. The same result occurs at batch edges, in an all-empty batch, and in a single-record batch. Non-empty arrays round-trip...
Getting duplicate records with offset pagination in Python
I'm pulling records from an API into BigQuery using Python. The API only has limit and offset, so I'm just fetching 100 records, adding 100 to the offset and doing the same thing until there are no more records.. This is the part I'm using:. offset = 0 records = [] while True: r...
Tired of messy dates and missing values? Here is my Python cleaning workflow.
Hey everyone,. I've been working with a lot of extremely messy real-world datasets lately (missing values, duplicate rows, completely mixed-up date formats like '2026.10.01' vs '10月1日', negative amounts). . Instead of just deleting rows (which ruins the data), I built a Python...
Why do enterprise analytics projects struggle even when the technology works?
...pipelines can be healthy, and dashboards can be available, yet business teams may still struggle to use the outputs confidently.. Sometimes the issue is inconsistent metric definitions. Sometimes it's unclear data ownership, a lack of trust in the numbers, or analytics that...
The order I check a messy export in, and why duplicates are last
...duplicate explosion from a bad join, duplicates jump to the front. But for the ordinary messy export, the duplicate rows are usually downstream of something simpler.. The part I'm less sure about: how much of this is worth automating versus checking by eye. Power Query can...
[v2 Bug] agate `Column.values_distinct()`, `values_without_nulls()`, `values_sorted()` and `values_without_nulls_sorted()` panic with `not yet implemented`
### Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x? - [x] I believe this is a new bug in dbt v2.x - [x] I have searched the existing issues and could not find a duplicate ### Current Behavior In dbt v2, calling `values_distinct()`,...
How do you check if your data is ready for AI before using it?
...or just check things manually?. And for those working with AI datasets, is this a real pain point for you or am I overthinking it?. Would love to hear how you guys handle this..
My SQL query returns the correct row count, but some records are still wrong. How do I find them?
I'm comparing a source table with a target table after an ETL job.. Both tables have the same row count, but I suspect some records are missing while others are duplicated or incorrectly updated.. The tables have millions of rows, no reliable primary key, and some columns...
[source-appsflyer] Incremental syncs of raw data reports permanently miss late-arriving events
### Connector Name source-appsflyer ### Connector Version 0.3.3 ### What step the error happened? During the sync ### Relevant information #### Summary Incremental syncs of the raw data streams can permanently skip events. Each sync resumes exactly at the saved cursor (the...
Is this SQL learning roadmap enough for real-world data analyst internships in 2026, or am I missing important skills?
...finish a bootcamp. I want to be able to solve unfamiliar business problems, work with messy data, validate results and explain findings.. I'm learning MySQL and using a SQL bootcamp that covers topics such as:. SELECT, WHERE, filtering and sorting. CASE statements and string...
How are you guys evaluating document parsers based on your documents? PDF parsing or whatever
...a plain check like - does this text appear check passes while the types in the repeated failing pattern can be thrown to something heavier like llamaparse for example or local stack like docling and other rust based ones. so each kind of doc goes to the cheapest thing that...
Classify preview outcomes and show error, no-data, and no-signal callouts
...grafana/grafana#134339), then rebase onto `main` once grafana/grafana#134339 merges. ## Blocked by * [Replace the Query Coauthoring session precedence chain with a reducer]( ) * [Introduce a cached, single-flight preview session for proposal options]( )
conversionPreservesOrder wrongly accepts Enum8 -> Bool as order-preserving
### Describe what's wrong - DISTINCT/GROUP BY over a Bool column mapped onto an Enum8 shard column returns duplicate rows instead of erroring. - Trigger: query a Distributed table whose column is declared Bool while the underlying MergeTree column is Enum8('a'=1,'b'=2). -...
What’s the smallest SQL mistake that caused the biggest problem for you???
Not talking about some crazy DB failure or anything.. Could be something silly like a missing WHERE, wrong JOIN, duplicate rows, bad UPDATE..... Like a small mistake that looked harmless but ended up causing a big headache ....🤕. Curious what ppl have seen in real projects..
ManualVuln7 information_schema.check_constraints Returns Duplicate Rows for Multi-Column CHECK Constraints
### What happens? `information_schema.check_constraints` (`src/catalog/default/default_views.cpp` line 213) is defined with a subquery that calls `UNNEST(constraint_column_names)` over rows from `duckdb_constraints()` filtered to `constraint_type IN ('CHECK', 'NOT NULL')`. For...
[BUG]KickoffTaskOutputsSQLiteStorage.update()/load() crash on non-dict values and NULL fields
### Description `KickoffTaskOutputsSQLiteStorage.update()` (lib/crewai/src/crewai/memory/storage/kickoff_task_outputs_storage.py:156) only JSON-encodes values when they're a `dict`: values.append( json.dumps(value, cls=CrewJSONEncoder) if isinstance(value, dict) else value )...
Replace duplicated DataFrame indices with a single Series
...dfdup.where(mask, "").max() In: dfresults Out: Sample1 Yellow Sample2 Yellow Sample3 Red How do I replace the 2 rows of df["A"] with the series dfresults without dropping duplicate rows (dropping duplicates from all of df would also affect "B" which hasn't been processed yet).....
[Spec][Parquet] Remove "Omit from data files" requirement for `UNKNOWN` type
...manifests: - STRUCT (this includes the schema root) with all fields omitted: there is no way to indicate the distinction between a row being `NULL` or `{}`. - LIST with omitted element: there is no way to indicate whether a list is `NULL` or `[]`. This has (presumably...
CustomWebhookNotificationBlock rejects JSON null values during initialization
### Bug summary `CustomWebhookNotificationBlock` cannot be constructed when `json_data` contains a JSON null (`None`), including nested object fields or list items. It raises `ValueError: Unexpected type: ` before any notification request is sent I reproduced this on the latest...
I Built a Data Quality Library Because Existing Tools Were Too Slow for My Pipelines
I got mass fed up with data quality tools, so I built my own I work as a data engineer....
find_orm_dags: multiple joinedload() collections cause row-explosion, slow DagModel sync (commonly >10s)
...fast enough. That points at result-set size, not query planning/execution cost, as the bottleneck. This customer reports these commonly exceeding 10 seconds. Response latency downstream (this call sits in the request path of the DAG-processor's parse-result heartbeat) has been...
Companies and products named
About this problem
Evidence
21 posts from 21 people in 18 places, about 2 a week over 64 days. Mostly on r/SQL, r/analytics, DEV #dataengineering. Tools named alongside: GitHub, Excel, pandas, DuckDB.
Frustration Frustration 1 of 3· Seen on GitHub issues, Reddit, Stack Overflow, DEV
How it was grouped
Posts that state a pain and match the "data-quality" rule. First post Sep 1, 2026, latest Nov 3, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Bad data slips through Databricks unnoticed
3 posts from 3 people on Reddit.
Bad data slips through Polars unnoticed
3 posts from 3 people on GitHub issues, Reddit.
Bad data slips through Snowflake unnoticed
3 posts from 3 people on Reddit, Hacker News.
Bad data slips through Apache Iceberg unnoticed
3 posts from 3 people on GitHub issues.
Bad data slips through Apache Spark unnoticed
2 posts from 2 people on GitHub issues. Unverified.
Bad data slips through PostgreSQL unnoticed
2 posts from 2 people on Stack Overflow. Unverified.
Bad data slips through Claude unnoticed
2 posts from 1 person on Reddit. Unverified.
History