Problemsby GlyphApache Airflow jobs fail silently or break often
13 posts from 12 people11d activeFading, +0 this week
Posts per day
The posts behind it
KubernetesJobOperator (deferrable): `get_logs=True` writes no pod logs when the Job fails
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.2 ### What happened and how to reproduce it? With `deferrable=True`, `wait_until_job_complete=True` and `get_logs=True`, a Job whose container exits non-zero fails the task with `Job...
fail_fast Dag with a task removed mid-run puts the scheduler into a crash loop
### Under which category would you file this issue? Airflow Core ### Apache Airflow version main (`d561f37ea5`, 2026-10-29); the same code path is present in 3.3.2 ### What happened and how to reproduce it? In a Dag with `fail_fast=True`, if a task is removed from the Dag...
@teardown task raises ValueError when called twice, and gets the wrong trigger rule with expand_kwargs
### Under which category would you file this issue? Task SDK ### Apache Airflow version 3.3.1 (also reproduced on `main` at 22f86cb551) ### What happened and how to reproduce it? A TaskFlow function decorated with `@teardown` can only be used once per Dag. Calling it a second...
@teardown task raises ValueError when called twice, and gets the wrong trigger rule with expand_kwargs
### Under which category would you file this issue? Task SDK ### Apache Airflow version 3.3.1 ### What happened and how to reproduce it? A TaskFlow function decorated with `@teardown` can only be used once per Dag. Calling it a second time, or calling `.override(...)` after the...
Add on_success_succeed_dagrun to teardown tasks
...gives the first line, and a failed teardown no longer fails the run. The workaround is to leave the flag off, so the failed work task still counts, and add a normal task after the teardown. An `EmptyOperator` with `trigger_rule=ALL_SUCCESS` is enough: a failed teardown marks it...
KubernetesStartKueueJobOperator: completion wait ignores Kueue preemption and requires the jobs/status subresource
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.1 ### What happened and how to reproduce it? `KubernetesStartKueueJobOperator` creates the Job suspended with the Kueue queue label, then inherits the waiting of `KubernetesJobOperator`....
API server event loop blocks on database queries while serializing get_dag_details and get_pools responses
## Under which category would you file this issue? Airflow Core ## Apache Airflow version 3.3.2 (the same code is on `main`) ## What happened and how to reproduce it? **Issue Description** Two response models of the public API query the metadata database during response...
SSHHook.exec_ssh_client_command fails with `ValueError: filedescriptor out of range in select()` when the task process holds >1024 file descriptors (hit easily under Airflow 3.x Task SDK)
...the local `select()` read path. (This refutes a commonly suggested "set get_pty=True" workaround.) - Running the same command via the system `ssh` client (which uses `poll()`), under the same high FD count, succeeds. ### Workaround We worked around this by replacing the...
CeleryExecutor task workload serialization fails with OverflowError for integers outside 64-bit range (msgspec msgpack)
### Apache Airflow version 3.x ### What happened? When the CeleryExecutor sends a task workload to the task-runner subprocess, Airflow serializes the message with `msgspec` (msgpack) in `airflow/sdk/execution_time/comms.py` (`as_bytes` -> `req_encoder.encode_into(...)`). If the...
Add link from calendar cell to filter by the given date as start_date
### Description When there is a dag failure in the calendar view for a given date then getting to the dagruns for the start date requires switching to the dagruns and then filtering by state. Since each cell is associated with a timestamp in case of daily or hourly runs clicking...
xcom_pull logs at ERROR level for an expected "XCom not found" case in Airflow 3.x Task SDK
...the same reason, though that's a separate call site - happy to split if preferred.) ### Workaround `[logging] namespace_levels = airflow.sdk.api.client=CRITICAL` in `airflow.cfg` silences it, but that's an all-or-nothing hammer on the whole client logger (also silences genuine...
KubernetesPodOperator task is marked SUCCESS when the task process receives SIGTERM mid-execute (pod deleted by on_kill, cleanup() short-circuits on _killed)
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.1 ### What happened and how to reproduce it? ### Issue Description When the task process running a `KubernetesPodOperator` receives `SIGTERM` while `execute()` is in flight, the task...
TriggerDagRunOperator fails with a 404 "Dag run with run_id: 'manual__2026-09-08T02:35:00+00:00' not found"
Using Airflow 3.3.1.. I am having trouble reproducing an error, but it's a consistent error.. Dag1 is responsible for fetching data from an external source, and saving a copy in a local cache. 1 dag run = 1 file in local cache. Many dags are dependent on this cache. The cache is...
Companies and products named
About this problem
Evidence
13 posts from 12 people in 2 places, about 2 a week over 56 days. Mostly on apache/airflow, Stack Overflow [airflow]. Tools named alongside: Kubernetes, GitHub, AWS, Docker.
Frustration Frustration 0 of 3· Seen on GitHub issues, Stack Overflow
How it was grouped
Posts that state a pain and match the "pipeline-failures" rule about Apache Airflow. First post Sep 8, 2026, latest Nov 2, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
The same problem elsewhere
Data pipelines fail silently or break often
19 posts from 19 people on GitHub issues, Reddit, Stack Overflow.
Gemini jobs fail silently or break often
8 posts from 1 person on GitHub issues. Unverified.
PostgreSQL jobs fail silently or break often
5 posts from 4 people on GitHub issues.
Databricks jobs fail silently or break often
5 posts from 5 people on GitHub issues, Reddit.
GitHub jobs fail silently or break often
4 posts from 4 people on Reddit, GitHub issues.
ChatGPT jobs fail silently or break often
3 posts from 3 people on GitHub issues, Community forums.
Claude Code jobs fail silently or break often
3 posts from 3 people on GitHub issues.
GitLab jobs fail silently or break often
2 posts from 2 people on Stack Overflow. Unverified.
Docker jobs fail silently or break often
2 posts from 2 people on GitHub issues. Unverified.
Other Apache Airflow problems
Apache Airflow users keep filing bugs
Top post: "Status of testing Providers that were prepared on October 20, 2026", one of 41 posts from 31 people on GitHub issues, Reddit.
Apache Airflow is too complex to run well
34 posts from 32 people on GitHub issues.
Apache Airflow upgrades break working setups
11 posts from 10 people on GitHub issues.
Debugging Apache Airflow failures is hard
7 posts from 6 people on GitHub issues.
Apache Airflow login, SSO or OAuth breaks
4 posts from 3 people on GitHub issues.
Apache Airflow backfills are slow and risky
4 posts from 4 people on GitHub issues.
Trouble with Apache Airflow retry
Top post: "Task state store writes from `on_retry_callback` fail with 404", one of 3 posts from 3 people on GitHub issues.
Trouble with Apache Airflow dags
Top post: "DagModel.owners falls back to 'airflow' for Dags with only mapped tasks (LazyDeserializedDAG.owner ignores", one of 3 posts from 3 people on GitHub issues.
Trouble with Apache Airflow backend
Top post: "Support [workers] state_store_backend in language SDK tasks", one of 3 posts from 3 people on GitHub issues.
Apache Airflow lock-in makes switching costly
2 posts from 2 people on GitHub issues. Unverified.
Apache Airflow runs out of memory
2 posts from 2 people on GitHub issues. Unverified.
Apache Airflow permissions are confusing and error-prone
2 posts from 2 people on GitHub issues. Unverified.
History