Problemsby GlyphApache Spark users keep filing bugs
16 posts from 15 people14d activeSteady, +2 this week
Posts per day
The posts behind it
Learned ZOrder Implementation on Parquet files using Apache Spark and Java
I am going through a research paper Learning Bit Allocations for Z-Order Layouts in Analytic Data Systems. https://openreview.net/forum?id=v3WOdLUnwL . I am trying to implement this paper in Java and Apache Spark on a Taxi dataset. What are the possible steps by which this can...
Structured Streaming Statestore is Not Working (Zerobus)
Zerobus is writing data to a landing table like so:. CREATE TABLE a.b.streaming_c ( MyA STRING NOT NULL, MyB INT NOT NULL, MyC STRING, MyDetails VARIANT, ZerobusRescueJson VARIANT) USING delta TBLPROPERTIES ( 'delta.checkpoint.writeStatsAsJson' = 'false',...
[PS] import pyspark.pandas fails with pandas 3.1.0 (DateOffset and infer_dtype_from_object no longer importable)
### What happens With pandas 3.1.0 (released 2026-10-22), `import pyspark.pandas` fails on PySpark 4.2.0 and on `master`: With that import fixed, the next one fails the same way: `pyspark[pandas_on_spark]` requires `pandas>=2.2.0` with no upper bound, so a fresh `pip install...
[SQL] SortMergeJoin codegen compilation fails when maxFields > 100 due to inner class variable scoping
### Describe the bug Raising `spark.sql.codegen.maxFields` above 100 to enable whole-stage codegen for wide schemas causes **SortMergeJoin codegen to fail compilation** on Spark master: The root cause: when generated code exceeds the class-size threshold, Spark's...
Did anyone here contribute to Apache Spark?
...or submitting PRs.. I’d love to hear about your experience, what you worked on, and any advice for someone who wants to start contributing..
[Build][Discuss] How do you do checking for license headers?
As a member of the Creadur PMC I just realised you do not use Apache Creadur's RAT plugin to check for license headers in your sources/the project. Are you interested in introducing RAT: https://creadur.apache.org/rat Thanks
[SPARK DECLARATIVE PIPELINES] public docs mention custom resource manager is compatible but a run fails with "Remote cannot be specified with master and/or deploy mode"
The spark public docs [here](https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html#the-spark-pipelines-command-line-interface) mention that > spark-pipelines is built on top of spark-submit, meaning that it supports all cluster managers supported by...
[BUG][Spark] Python DeltaTable.clone, cloneAtVersion and cloneAtTimestamp return a py4j JavaObject instead of a DeltaTable
## Bug #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ### Describe the problem The Python `DeltaTable.clone`, `cloneAtVersion` and `cloneAtTimestamp` are documented and annotated to return a...
Apache Spark, PySpark can't write a file on disk
...running docker-compose script from Windows command line as admin user, but it still fails to create "temporary directories"..
Is a broadcast variable shared across multiple concurrent Spark jobs?
I have a PySpark application where I use ThreadPoolExecutor to process multiple independent workloads concurrently.. Each thread performs a Spark join using the same dimension DataFrame:. def process_one(source): fact = ... result = fact.join( F.broadcast(dim), join_condition,...
index.html
Classroom - Gaming Platform * { margin: 0; padding: 0; box-sizing: border-box; } :root { --bg-primary: #0a0e27; --bg-secondary: #1a1f3a; --bg-tertiary: #252d4a; --text-primary: #ffffff; --text-secondary: #b0b9cc; --accent-primary: #7c3aed; --accent-secondary: #ec4899;...
packaging data products
Hi, how do u handle data products, and chain of a lot of transformations, it can be very messy in the notebooks as mix of sqls, sqls python strings, python code, pyspark code, if/elses, etc.... But again, is it really worth it to modularize it, make it callable, testable. I find...
The feeling of not knowing what you are doing (sometimes) comes from not writing your own functions
...got a take home where the goal was to use duck db and do a bread and butter pipeline. Wa struggling since pyspark is my main tool for some years now.. Like, how i am supposed to parse a date column with various formats. You would keep checking the documentation and search there...
[SQL] Row-based Parquet reader silently accepts incompatible primitive type conversions
...because changing `spark.sql.parquet.enableVectorizedReader` can change a query from failing cleanly to returning incorrectly interpreted data. ### Expected behavior Both readers should reject these unsupported conversions with...
Adding delays between JDBC connection retries
...Postgres deployments for usage with AGE etc., all read/write via Spark. We have an issue with connection retries during RDS rollovers or other intermittent issues. For writers this is not a problem as we can add our own delay with backoff and then retry the writer. However, For...
Credential Rotation
Hello, In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark and spark-connect. We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark. We would like to...
Companies and products named
About this problem
Evidence
16 posts from 15 people in 6 places, about 2 a week over 66 days. Mostly on apache/spark, r/databricks, Stack Overflow [apache-spark]. Tools named alongside: Databricks, PostgreSQL, AWS, PyPI.
Frustration Frustration 0 of 3· Seen on GitHub issues, Reddit, Stack Overflow
How it was grouped
Posts about Apache Spark that state a pain but match no known issue yet. First post Sep 2, 2026, latest Nov 6, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
Other Apache Spark problems
Apache Spark upgrades break working setups
4 posts from 4 people on GitHub issues.
Trouble with Apache Spark rows
Top post: "scan_delta returns deleted rows from partitions whose path is percent-encoded", one of 4 posts from 4 people on GitHub issues, Stack Overflow.
Bad data slips through Apache Spark unnoticed
2 posts from 2 people on GitHub issues. Unverified.
History