Databricks jobs fail silently or break often

Data engineering2 posts from 2 people2d active+11 post in the last 7 days, 0 the 7 days before (steady)Unverified

Posts per day2 posts, Aug 27 to Sep 25

The posts behind it

2, newest first

Post	Where	Date
Conditionally retrying a task on Lakeflow Jobs...I've been diging into docs and forums, but I cannot find a native approach or non-patchy workaround to only retry on certain conditions (like based on exit codes).. So far, I've come with these candidate solutions:. - Just retrying everything (which doesn't makes sense cause if...r/databricksu/Xandor19	r/databricksu/Xandor19	Sep 24yesterday
External jobs are never cancelled when a task fails terminally### Summary An operator submits a job to an external system (Spark, Glue, Databricks, BigQuery) and polls it. If the task fails without being killed, and has no retries left, nothing cancels that external job. The Dag run finishes, the task is red, and the job keeps consuming...apache/airflowamoghrajesh	apache/airflowamoghrajesh	Aug 274 weeks ago

Companies and products named

Company or product	Posts naming it
Databricks	2
BigQuery	1
Apache Airflow	1

About this problem

Evidence

2 posts from 2 people in 2 places, about 1 a week over 29 days. Mostly on apache/airflow, r/databricks. Tools named alongside: BigQuery, Apache Airflow.

Frustration Frustration 0 of 3· Seen on GitHub issues, Reddit

How it was grouped

Posts that state a pain and match the "pipeline-failures" rule about Databricks. First post Aug 27, 2026, latest Sep 24, 2026. Unverified: fewer than 3 posts, or one person or place only. Method

The same problem elsewhere

Posts per day

The posts behind it

Companies and products named

Rising problems by email