Apache Spark users keep filing bugs

Data engineering5 posts from 4 people4d active+11 post in the last 7 days, 0 the 7 days before (steady)

Posts per day5 posts, Sep 2 to Sep 25

The posts behind it

5, newest first

Post	Where	Date
packaging data productsHi, how do u handle data products, and chain of a lot of transformations, it can be very messy in the notebooks as mix of sqls, sqls python strings, python code, pyspark code, if/elses, etc.... But again, is it really worth it to modularize it, make it callable, testable. I find...r/databricksu/ptab0211	r/databricksu/ptab0211	Sep 223 days ago
The feeling of not knowing what you are doing (sometimes) comes from not writing your own functions...got a take home where the goal was to use duck db and do a bread and butter pipeline. Wa struggling since pyspark is my main tool for some years now.. Like, how i am supposed to parse a date column with various formats. You would keep checking the documentation and search there...r/dataengineeringu/Old_Tourist_3774	r/dataengineeringu/Old_Tourist_3774	Sep 43 weeks ago
[SQL] Row-based Parquet reader silently accepts incompatible primitive type conversions...because changing `spark.sql.parquet.enableVectorizedReader` can change a query from failing cleanly to returning incorrectly interpreted data. ### Expected behavior Both readers should reject these unsupported conversions with...apache/sparkJiayi-Wang-db	apache/sparkJiayi-Wang-db	Sep 33 weeks ago
Adding delays between JDBC connection retries...Postgres deployments for usage with AGE etc., all read/write via Spark. We have an issue with connection retries during RDS rollovers or other intermittent issues. For writers this is not a problem as we can add our own delay with backoff and then retry the writer. However, For...apache/sparktmaoz	apache/sparktmaoz1 reactions	Sep 23 weeks ago
Credential RotationHello, In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark and spark-connect. We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark. We would like to...apache/sparktmaoz	apache/sparktmaoz1 replies	Sep 23 weeks ago

Companies and products named

Company or product	Posts naming it
Apache Spark	5
PostgreSQL	2
AWS	2
DuckDB	1
Databricks	1

About this problem

Evidence

5 posts from 4 people in 3 places, about 2 a week over 21 days. Mostly on apache/spark, r/dataengineering, r/databricks. Tools named alongside: PostgreSQL, AWS, DuckDB, Databricks.

Frustration Frustration 0 of 3· Seen on GitHub issues, Reddit

How it was grouped

Posts about Apache Spark that state a pain but match no known issue yet. First post Sep 2, 2026, latest Sep 22, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

Posts per day

The posts behind it

Companies and products named

Rising problems by email