Data engineering problems

54 problems, rising first. The list grows every day.

Problem	Companies	Active	Seen on	This week
Bad data slips into production unnoticedApache Iceberg, Snowflake, Excel, pandas, Apache Airflow, PostgreSQL7d active+1Seen on GitHub issues, DEV, Reddit, Stack Overflow	Apache Iceberg, Snowflake, Excel, pandas, Apache Airflow, PostgreSQL	7d activesince Sep 1	Seen on GitHub issues, DEV, Reddit, Stack Overflow	+33 posts in the last 7 days, 0 the 7 days before (rising)
Dagster backfills are slow and riskyDagster, PostgreSQL, Docker3d activeSeen on GitHub issues	Dagster, PostgreSQL, Docker	3d activesince Sep 21	Seen on GitHub issues	+33 posts in the last 7 days, 0 the 7 days before (new)
Small files, skew and partitioning hurt performanceGitHub, Apache Iceberg, llama.cpp, Claude Code, Microsoft Azure, Polars9d activeSeen on GitHub issues, Reddit	GitHub, Apache Iceberg, llama.cpp, Claude Code, Microsoft Azure, Polars	9d activesince Aug 28	Seen on GitHub issues, Reddit	+33 posts in the last 7 days, 0 the 7 days before (rising)
Hard to trace lineage and find trusted tablesDagster, Apache Spark, Anthropic, OpenAI, Perplexity, DeepSeek4d activeSeen on GitHub issues	Dagster, Apache Spark, Anthropic, OpenAI, Perplexity, DeepSeek	4d activesince Sep 1	Seen on GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
Migrating between platforms is slow and riskyNotion, Kubernetes, Elasticsearch, vLLM, Hugging Face, GitHub8d active+2Seen on Reddit, GitHub issues, App Store reviews, Hacker News, Community forums	Notion, Kubernetes, Elasticsearch, vLLM, Hugging Face, GitHub	8d activesince Aug 31	+1Seen on Reddit, GitHub issues, App Store reviews, Hacker News, Community forums	+33 posts in the last 7 days, 2 the 7 days before (steady)
Snowflake users keep asking for helpSnowflake, dbt, Apache Spark, Apache Airflow, AWS, Google Cloud5d activeSeen on Stack Overflow, Reddit, GitHub issues	Snowflake, dbt, Apache Spark, Apache Airflow, AWS, Google Cloud	5d activesince Aug 27	Seen on Stack Overflow, Reddit, GitHub issues	+44 posts in the last 7 days, 0 the 7 days before (rising)
Data pipelines fail silently or break oftenGitHub, ChatGPT, OpenAI, Gemini, Prefect, Slack3d activeSeen on GitHub issues, Community forums	GitHub, ChatGPT, OpenAI, Gemini, Prefect, Slack	3d activesince Sep 3	Seen on GitHub issues, Community forums	+22 posts in the last 7 days, 0 the 7 days before (rising)
Bad data slips through Databricks unnoticedDatabricks, Microsoft Azure, GitHub1d activeSeen on RedditUnverified	Databricks, Microsoft Azure, GitHub	1d activesince Sep 22	Seen on Reddit	+22 posts in the last 7 days, 0 the 7 days before (new)
Apache Kafka users keep filing bugsApache Kafka, Apache Iceberg, GitHub, Confluent, Apache Flink, Docker4d activeSeen on GitHub issues, Stack Overflow	Apache Kafka, Apache Iceberg, GitHub, Confluent, Apache Flink, Docker	4d activesince Sep 2	Seen on GitHub issues, Stack Overflow	+33 posts in the last 7 days, 0 the 7 days before (rising)
Trouble with Delta Lake protocolDelta Lake, GitHub, IEEE4d activeSeen on GitHub issues	Delta Lake, GitHub, IEEE	4d activesince Aug 31	Seen on GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
Prefect users keep filing bugsPrefect, GitHub, Google Cloud, Nebius, Kubernetes9d activeSeen on GitHub issues	Prefect, GitHub, Google Cloud, Nebius, Kubernetes	9d activesince Aug 27	Seen on GitHub issues	+44 posts in the last 7 days, 2 the 7 days before (rising)
Apache Flink users keep filing bugsApache Flink, Delta Lake, Apache Iceberg, GitHub5d activeSeen on GitHub issues, Reddit	Apache Flink, Delta Lake, Apache Iceberg, GitHub	5d activesince Aug 28	Seen on GitHub issues, Reddit	+22 posts in the last 7 days, 0 the 7 days before (rising)
Databricks users keep asking for helpDatabricks, Apache Flink, Delta Lake, Microsoft Azure, Azure Data Factory7d activeSeen on Reddit, GitHub issues	Databricks, Apache Flink, Delta Lake, Microsoft Azure, Azure Data Factory	7d activesince Sep 1	Seen on Reddit, GitHub issues	+33 posts in the last 7 days, 1 the 7 days before (rising)
Apache Iceberg users keep filing bugsApache Iceberg, GitHub, Amazon S3, Docker7d activeSeen on GitHub issues	Apache Iceberg, GitHub, Amazon S3, Docker	7d activesince Aug 30	Seen on GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
dbt users keep asking for helpdbt, Dagster, DuckDB4d activeSeen on Community forums, Reddit, GitHub issues	dbt, Dagster, DuckDB	4d activesince Sep 4	Seen on Community forums, Reddit, GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
pandas users keep filing bugspandas, GitHub, PostgreSQL, SQLite, Prefect11d activeSeen on GitHub issues, Stack Overflow	pandas, GitHub, PostgreSQL, SQLite, Prefect	11d activesince Aug 27	Seen on GitHub issues, Stack Overflow	+33 posts in the last 7 days, 2 the 7 days before (steady)
Airbyte users keep filing bugsAirbyte, GitHub, Kubernetes, Google Cloud, Apache Superset, AWS8d activeSeen on GitHub issues	Airbyte, GitHub, Kubernetes, Google Cloud, Apache Superset, AWS	8d activesince Sep 1	Seen on GitHub issues	+22 posts in the last 7 days, 3 the 7 days before (steady)
Backfills are slow, costly and riskyGitHub, Snowflake, Azure Data Factory, SQL Server, Anthropic, OpenAI5d activeSeen on GitHub issues, Reddit	GitHub, Snowflake, Azure Data Factory, SQL Server, Anthropic, OpenAI	5d activesince Sep 9	Seen on GitHub issues, Reddit	+11 post in the last 7 days, 3 the 7 days before (fading)
Trouble with Prefect flowPrefect, GitHub5d activeSeen on GitHub issues	Prefect, GitHub	5d activesince Aug 28	Seen on GitHub issues	+11 post in the last 7 days, 2 the 7 days before (steady)
Trouble with pandas seriespandas, PostgreSQL, SQLite, GitHub4d activeSeen on GitHub issues, Stack Overflow	pandas, PostgreSQL, SQLite, GitHub	4d activesince Aug 27	Seen on GitHub issues, Stack Overflow	+11 post in the last 7 days, 0 the 7 days before (steady)
Apache Spark users keep filing bugsApache Spark, PostgreSQL, AWS, DuckDB, Databricks4d activeSeen on GitHub issues, Reddit	Apache Spark, PostgreSQL, AWS, DuckDB, Databricks	4d activesince Sep 2	Seen on GitHub issues, Reddit	+11 post in the last 7 days, 0 the 7 days before (steady)
Apache Airflow jobs fail silently or break oftenApache Airflow, GitHub, AWS, Kubernetes4d activeSeen on GitHub issues, Stack Overflow	Apache Airflow, GitHub, AWS, Kubernetes	4d activesince Sep 8	Seen on GitHub issues, Stack Overflow	+11 post in the last 7 days, 1 the 7 days before (steady)
Hard to trace lineage and find trusted data in GitHubGitHub, VS Code, Apache Iceberg, Kubernetes3d activeSeen on GitHub issues	GitHub, VS Code, Apache Iceberg, Kubernetes	3d activesince Sep 3	Seen on GitHub issues	+11 post in the last 7 days, 0 the 7 days before (steady)
Bad data slips through Polars unnoticedPolars2d activeSeen on Reddit, GitHub issuesUnverified	Polars	2d activesince Sep 4	Seen on Reddit, GitHub issues	+11 post in the last 7 days, 0 the 7 days before (steady)
Databricks jobs fail silently or break oftenDatabricks, BigQuery, Apache Airflow2d activeSeen on GitHub issues, RedditUnverified	Databricks, BigQuery, Apache Airflow	2d activesince Aug 27	Seen on GitHub issues, Reddit	+11 post in the last 7 days, 0 the 7 days before (steady)
Apache Iceberg schema changes break downstream pipelinesApache Iceberg, Airbyte, ClickHouse, PostgreSQL, MySQL, SQL Server2d activeSeen on GitHub issuesUnverified	Apache Iceberg, Airbyte, ClickHouse, PostgreSQL, MySQL, SQL Server	2d activesince Sep 18	Seen on GitHub issues	+11 post in the last 7 days, 1 the 7 days before (steady)
Data modeling choices are unclear and hard to changedbt2d activeSeen on RedditUnverified	dbt	2d activesince Sep 9	Seen on Reddit	+11 post in the last 7 days, 0 the 7 days before (steady)
GitLab jobs fail silently or break oftenGitLab, Docker, GitHub2d activeSeen on Stack OverflowUnverified	GitLab, Docker, GitHub	2d activesince Aug 26	Seen on Stack Overflow	+11 post in the last 7 days, 0 the 7 days before (steady)
AWS Glue users keep filing bugsAWS Glue, Apache Iceberg2d activeSeen on GitHub issuesUnverified	AWS Glue, Apache Iceberg	2d activesince Aug 28	Seen on GitHub issues	+11 post in the last 7 days, 0 the 7 days before (steady)
Apache Airflow users keep filing bugsApache Airflow, GitHub, Claude Code, dbt, PostgreSQL, Apache Kafka5d activeSeen on GitHub issues, Reddit	Apache Airflow, GitHub, Claude Code, dbt, PostgreSQL, Apache Kafka	5d activesince Sep 1	Seen on GitHub issues, Reddit	+00 posts in the last 7 days, 0 the 7 days before (fading)
Polars users keep filing bugsPolars, GitHub, PyPI7d activeSeen on GitHub issues	Polars, GitHub, PyPI	7d activesince Aug 27	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
Trouble with Polars columnPolars, PyPI, GitHub4d activeSeen on GitHub issues	Polars, PyPI, GitHub	4d activesince Aug 30	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
Migrating to or off GitHub is slow and riskyGitHub, VS Code, Apache Iceberg, Grafana, Vercel4d activeSeen on GitHub issues	GitHub, VS Code, Apache Iceberg, Grafana, Vercel	4d activesince Sep 2	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
BigQuery users keep asking for helpBigQuery, GitHub, Apache Spark, Docker, Claude Code, Apache Airflow4d activeSeen on Stack Overflow, GitHub issues	BigQuery, GitHub, Apache Spark, Docker, Claude Code, Apache Airflow	4d activesince Aug 30	Seen on Stack Overflow, GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
MySQL connectors and syncs are fragileMySQL, Airbyte, GitHub, AWS, Apache Kafka, Snowflake4d activeSeen on GitHub issues	MySQL, Airbyte, GitHub, AWS, Apache Kafka, Snowflake	4d activesince Aug 28	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Dagster users keep filing bugsDagster, PostgreSQL, Slack, Docker, Kubernetes4d activeSeen on GitHub issues	Dagster, PostgreSQL, Slack, Docker, Kubernetes	4d activesince Sep 1	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with pandas testpandas2d activeSeen on GitHub issues	pandas	2d activesince Sep 4	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
Ingestion connectors and CDC are fragileAirbyte, MySQL, AWS, MongoDB, Apache Iceberg3d activeSeen on GitHub issues	Airbyte, MySQL, AWS, MongoDB, Apache Iceberg	3d activesince Sep 4	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
PostgreSQL connectors and syncs are fragilePostgreSQL, Airbyte, GitHub, pandas, Stack Overflow3d activeSeen on GitHub issues	PostgreSQL, Airbyte, GitHub, pandas, Stack Overflow	3d activesince Aug 28	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with pandas insteadpandas, PostgreSQL, SQLite, GitHub3d activeSeen on GitHub issues	pandas, PostgreSQL, SQLite, GitHub	3d activesince Aug 28	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with Polars schemaPolars, PyPI2d activeSeen on GitHub issues	Polars, PyPI	2d activesince Aug 29	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with pandas inputpandas, GitHub2d activeSeen on Stack Overflow, GitHub issuesUnverified	pandas, GitHub	2d activesince Sep 4	Seen on Stack Overflow, GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
Debugging Apache Airflow failures is hardApache Airflow, GitHub, Kubernetes, Docker, AWS2d activeSeen on GitHub issuesUnverified	Apache Airflow, GitHub, Kubernetes, Docker, AWS	2d activesince Sep 4	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
Delta Lake users keep filing bugsDelta Lake, Apache Iceberg2d activeSeen on GitHub issuesUnverified	Delta Lake, Apache Iceberg	2d activesince Sep 8	Seen on GitHub issues	+00 posts in the last 7 days, 1 the 7 days before (fading)
GitHub jobs fail silently or break oftenGitHub, Airbyte2d activeSeen on Reddit, GitHub issuesUnverified	GitHub, Airbyte	2d activesince Sep 1	Seen on Reddit, GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Airbyte schema changes break downstream pipelinesAirbyte, PostgreSQL2d activeSeen on GitHub issuesUnverified	Airbyte, PostgreSQL	2d activesince Sep 9	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with Polars joinPolars, PyPI2d activeSeen on GitHub issuesUnverified	Polars, PyPI	2d activesince Aug 29	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with Polars rollingPolars, PyPI2d activeSeen on GitHub issuesUnverified	Polars, PyPI	2d activesince Sep 5	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Trouble with Polars insteadPolars, PyPI2d activeSeen on GitHub issuesUnverified	Polars, PyPI	2d activesince Aug 28	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)
Apache Airflow backfills are slow and riskyApache Airflow, GitHub, Kubernetes, Docker2d activeSeen on GitHub issuesUnverified	Apache Airflow, GitHub, Kubernetes, Docker	2d activesince Aug 31	Seen on GitHub issues	+00 posts in the last 7 days, 0 the 7 days before (fading)

Data engineering problems

Rising problems by email