Data engineering problems

12 problems, rising first. The list grows every day.

Problem	Companies	Active	Seen on	This week
Bad data slips into production unnoticedApache Iceberg, Snowflake, Excel, pandas, Apache Airflow, PostgreSQL7d active+1Seen on GitHub issues, DEV, Reddit, Stack Overflow	Apache Iceberg, Snowflake, Excel, pandas, Apache Airflow, PostgreSQL	7d activesince Sep 1	Seen on GitHub issues, DEV, Reddit, Stack Overflow	+33 posts in the last 7 days, 0 the 7 days before (rising)
Small files, skew and partitioning hurt performanceGitHub, Apache Iceberg, llama.cpp, Claude Code, Microsoft Azure, Polars9d activeSeen on GitHub issues, Reddit	GitHub, Apache Iceberg, llama.cpp, Claude Code, Microsoft Azure, Polars	9d activesince Aug 28	Seen on GitHub issues, Reddit	+33 posts in the last 7 days, 0 the 7 days before (rising)
Hard to trace lineage and find trusted tablesDagster, Apache Spark, Anthropic, OpenAI, Perplexity, DeepSeek4d activeSeen on GitHub issues	Dagster, Apache Spark, Anthropic, OpenAI, Perplexity, DeepSeek	4d activesince Sep 1	Seen on GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
Snowflake users keep asking for helpSnowflake, dbt, Apache Spark, Apache Airflow, AWS, Google Cloud5d activeSeen on Stack Overflow, Reddit, GitHub issues	Snowflake, dbt, Apache Spark, Apache Airflow, AWS, Google Cloud	5d activesince Aug 27	Seen on Stack Overflow, Reddit, GitHub issues	+44 posts in the last 7 days, 0 the 7 days before (rising)
Data pipelines fail silently or break oftenGitHub, ChatGPT, OpenAI, Gemini, Prefect, Slack3d activeSeen on GitHub issues, Community forums	GitHub, ChatGPT, OpenAI, Gemini, Prefect, Slack	3d activesince Sep 3	Seen on GitHub issues, Community forums	+22 posts in the last 7 days, 0 the 7 days before (rising)
Apache Kafka users keep filing bugsApache Kafka, Apache Iceberg, GitHub, Confluent, Apache Flink, Docker4d activeSeen on GitHub issues, Stack Overflow	Apache Kafka, Apache Iceberg, GitHub, Confluent, Apache Flink, Docker	4d activesince Sep 2	Seen on GitHub issues, Stack Overflow	+33 posts in the last 7 days, 0 the 7 days before (rising)
Trouble with Delta Lake protocolDelta Lake, GitHub, IEEE4d activeSeen on GitHub issues	Delta Lake, GitHub, IEEE	4d activesince Aug 31	Seen on GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
Prefect users keep filing bugsPrefect, GitHub, Google Cloud, Nebius, Kubernetes9d activeSeen on GitHub issues	Prefect, GitHub, Google Cloud, Nebius, Kubernetes	9d activesince Aug 27	Seen on GitHub issues	+44 posts in the last 7 days, 2 the 7 days before (rising)
Apache Flink users keep filing bugsApache Flink, Delta Lake, Apache Iceberg, GitHub5d activeSeen on GitHub issues, Reddit	Apache Flink, Delta Lake, Apache Iceberg, GitHub	5d activesince Aug 28	Seen on GitHub issues, Reddit	+22 posts in the last 7 days, 0 the 7 days before (rising)
Databricks users keep asking for helpDatabricks, Apache Flink, Delta Lake, Microsoft Azure, Azure Data Factory7d activeSeen on Reddit, GitHub issues	Databricks, Apache Flink, Delta Lake, Microsoft Azure, Azure Data Factory	7d activesince Sep 1	Seen on Reddit, GitHub issues	+33 posts in the last 7 days, 1 the 7 days before (rising)
Apache Iceberg users keep filing bugsApache Iceberg, GitHub, Amazon S3, Docker7d activeSeen on GitHub issues	Apache Iceberg, GitHub, Amazon S3, Docker	7d activesince Aug 30	Seen on GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)
dbt users keep asking for helpdbt, Dagster, DuckDB4d activeSeen on Community forums, Reddit, GitHub issues	dbt, Dagster, DuckDB	4d activesince Sep 4	Seen on Community forums, Reddit, GitHub issues	+22 posts in the last 7 days, 0 the 7 days before (rising)

Data engineering problems

Rising problems by email