Problemsby GlyphApache Airflow users keep filing bugs
41 posts from 31 people22d activeFading, +2 this week
Posts per day
The posts behind it
Need advice on upgrading homelab/self-hosting computer
I've recently gotten excited about the idea of self-hosting apps and so got my hands on an old OptiPlex 3070 SFF with no modifications atm. I've been setting things up nicely with Ubuntu Server, docker and various containers, and relying on an older Seagate Ironwolf NAS drive to...
Iceberg: add IcebergHook.append() for writing rows to a table
...aligned to the Iceberg table schema (keys not in the schema are dropped instead of failing the write). - On `CommitFailedException`, reloads the table from the catalog and retries a bounded number of times. - Does not accept a `RecordBatchReader`: it is consumed on the first...
Support TLS connections to log-servers
### Description Currently connections to fetch the logs from e.g. the workers are done unencrypted via HTTP. In other parts of airflow you can already specify that TLS connections should be used and there are config options to provide corresponding certificates and keys. For...
Prompt user to confirm when clicking to pause a Dag
### Description Presently, pausing a Dag in the UI is a frictionless experience, allowing users to inadvertently break critical pipelines. This proposal is to add a modal/ pop-up when the user clicks on the pause/activate toggle, asking them to confirm/ cancel their request to...
Executor callbacks get global env var secrets instead of their team's in multi-team mode
### What happened In multi-team mode, a callback run by an executor (for example a deadline callback) that calls `Variable.get("my_var")` or `Connection.get("my_conn")` gets the global `AIRFLOW_VAR_MY_VAR` / `AIRFLOW_CONN_MY_CONN`, even when its team has `AIRFLOW_VAR__...
Editing a Variable in the UI overwrites masked JSON fields with ***
### Apache Airflow version 3.2.2, also reproduced on main (8401a190fa). The variable PATCH path has no merge step in any 3.x release (3.0.0 through 3.4.0rc1). ### What happened? `VariableResponse.redact_val` masks sensitive keys at any depth of a JSON Variable value, so GET...
Feature Request: Support for Fragment URIs in Assets
### Description I'm working with a data orchestration pipeline using Apache Airflow where DAGs are triggered by Assets that correspond to RDF vocabulary terms. Our assets use fragment URIs (e.g., https://example.org/vocab#JobA, https://example.org/vocab#JobB) to align with an...
How to get task-level observability for dbt models in Airflow without killing performance?
...and using Astronomer Cosmos. Both map models to Airflow tasks nicely but have terrible execution overhead. On the other hand, running a single dbt run is fast but gives zero observability in the Airflow UI.. How do you guys map dbt models/snapshots as individual Airflow tasks...
start_from_trigger do not respect airflow pools
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.3.2 ### What happened and how to reproduce it? the scheduler do not respect airflow pools when start_from_trigger=True ### What you think should happen instead? scheduler should respect...
previously_succeeded metric is documented but never emitted in Airflow 3
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.3.2; also reproduced on main on October 21, 2026 (3.4.0b2 has the same code). ### What happened and how to reproduce it? The metrics docs list the counter `previously_succeeded`...
Rendered Templates tab is empty when a task runs entirely in the triggerer (start_from_trigger + end_from_trigger)
### Under which category would you file this issue? Providers ### Apache Airflow version 3.3.2 ### What happened and how to reproduce it? ### What you think should happen instead? The triggerer should register the info , so user can see it in the airflow UI ### Versions of...
Implement Python task_handler decorator for YAML Dag ‘run’ tasks
...language SDKs (currently Go, Java, TypeScript) through AIP-108, but there is currently no way to implement a `run:` task's body in Python. This proposes creating an interface in Python equivalent to AIP-108 task handlers. This would allow Dag-authroing interfaces in other...
Status of testing Providers that were prepared on October 20, 2026
...@amoghrajesh - [ ] [Raise a clear error when SqlToS3Operator has no Parquet engine (#74817)](https://github.com/apache/airflow/pull/74817): @Dev-iL Linked issues: - [Linked Issue #74691](https://github.com/apache/airflow/pull/74691): @Dev-iL - [ ] [Fix SqlToS3Operator failing...
DynamicRegion.get_or_create_many() breaks scheduler-created DagRuns for dynamically mapped tasks (UNEXPECTED COMMIT - THIS WILL BREAK HA LOCKS!)
...treats the same as a real commit and rejects - it has no way to distinguish a SAVEPOINT release from an actual commit. This is the exact same `CommitProhibitorGuard` vs. nested-transaction mechanics already documented at length in #59120, but a **different** code path: #59120 is...
DbtCloudHook ignores timeout_seconds and connection proxies on sync requests
### Under which category would you file this issue? Providers ### Apache Airflow version 3.1.x (also reproduced on main at eee6e2a4a5) ### What happened and how to reproduce it? `DbtCloudHook(timeout_seconds=...)` has no effect on the sync API calls. The request goes out with...
EcsRunTaskOperator fails with ResourceNotFoundException instead of its own error or skip when the log stream is missing
...else? `BatchOperator` calls the same method inside a `try` that logs a warning ("Unable to retrieve CloudWatch logs for AWS Batch job"), so it is not affected; the ECS failure path could do the same. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ###...
Clicking on looped TI in grid view yields 400 errors
Reported Version: 3.4.0b1 RIght now if you click on the Grid view for a looped task, the API rejects it as the dag_id/run_id/task_id/map_index coordinates no longer uniquely identify a task when loops are involved. We need to decide what to show when you click on a looped TI in...
Go SDK: Add `--strict` flag for packer
### Body - refer to https://github.com/apache/airflow/pull/74637#discussion_r4259077768 - Add a `--strict` flag for the packer and fail the packing if there isn't any Dag ### Committer - [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Add support for rescheduling tasks to RetryPolicy
### Description The [RetryDecision](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/definitions/retry_policy.py#L71) interface for task policies does not support Rescheduling as an option. In our KubernetesPodOperator tasks we Reschedule the task if a...
Make migration 0142 (TI/TIH merge) downgrade safe for simple cases
Right now this migration will refuse to work if _any_ data has been written to xcom_v2. But we can make it work without data loss (or no more than would happen "natural for retries" in 3.3) as long as no task loops have run. > I'll double check/rethink if the downgrade works...
KeycloakAuthManager authorization conflict with Keycloak resource server Decision Strategy
### Under which category would you file this issue? Providers ### Apache Airflow version 3.2.2 ### What happened and how to reproduce it? I am using Apache Airflow 3.2.2 with apache-airflow-providers-keycloak 0.11.0 and Keycloak 26.4.7. Authentication works correctly. Users can...
Support optional DagID on the `get_source_code` signature for better UI code view visibility
### Body - For the DagImporter base class, support the optinal kwarg `dag_id` - For rest of the Language DagImporter (Node, Java) they can opt-in the `dag_id` and show the "source where it define the Dag" instead of always showing the entrypoint file - refer to...
UI: asset schedule count and layout depend on which assets the caller may read
### Description Since #72864 and #72862, `GET /ui/next_run_assets/{dag_id}` returns only the scheduling assets the caller may read, while `asset_expression` keeps a hidden slot for each unreadable one. `AssetSchedule.tsx` still derives both the total and the layout from the...
"match anywhere" in Dag search UI should be persisted as URL param
### Description We have the ability to search for Dags by display-name/ ID. By default, the search looks at the start of the Dag_id, and we have the option to turn on "match anywhere", to search for a pattern anywhere in the name of the Dag. However, this property is not...
README quickstart pins apache-airflow==3.3.0 while latest release is 3.3.2
## Finding The README quickstart on `main` pins the quickstart install to `apache-airflow==3.3.0`, but the newest published release is **3.3.2** (2026-09-17) and `main`'s `pyproject.toml` declares **3.4.0**. The quickstart therefore installs a two-releases-old version while...
Eager-load DagCode in the Dag source API
### Description `GET /dags/{dag_id}/dagSources` loads the `DagVersion` first. The `dag_code` relationship is lazy, so reading `dag_version.dag_code` sends a second query. Loading `dag_code` together with the version (e.g. via `joinedload`) would avoid that extra database round...
Clearing a task instance with both "Upstream" and "Downstream" clears unrelated tasks (whole run when the DAG has a shared root)
...original seed set and union the results, plus a test that enables both flags together. Workaround: clear twice, once with Upstream only and once with Downstream only. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to...
merge_dicts raises AttributeError when dict2's value is a dict but dict1's value for the same key is not
`airflow.utils.helpers.merge_dicts` documents: "Items in dict2 overwrite those also found in dict1." But when a key exists in both dicts and dict2's value is a dict while dict1's value is a scalar, it recurses unconditionally: Line 177 checks `isinstance(v, dict)` but never...
Development Environment for ML/MLOps
...different tools such as Kubeflow, Airflow, Evidently etc.. However, it can be really painful to install or to go back and forth between tools.. I was wondering if there any platform that provide throw away/development environments with ready to use tools ? A bit like GitHub...
BaseMessageQueueProvider declares abstractmethods without ABC, and three in-tree providers do not implement them
...dispatch makes the provider silently never match by queue instead of failing loudly. **2. Three in-tree providers rely on that gap.** `PubSubMessageQueueEventTriggerContainer` (google), `RedisPubSubMessageQueueProvider` (redis), and `AzureServiceBusMessageQueueProvider`...
How can I run my airflow pipeline continuously.
...Processer) but I don't want to start this process manually everytime so does anyone know how can I keep pipeline running continuously which ingest new data in database manually triggering anything.. Does anyone have a any suggetion .
DataflowJobStatusTrigger - Add param job_name
### Description Add support for filtering/monitoring Dataflow jobs by `job_name` in `DataflowJobStatusSensor` (and its corresponding trigger `DataflowJobStatusTrigger`), rather than requiring `job_id`. Currently, `DataflowJobStatusSensor` strictly requires `job_id`. Because...
Multi-Team: Allow creating team unique resources
### Description Multi team deployments use the same uniqueness constraints as regular airflow deployments, including dag_id, pool_id, variable_id and conn_id needing to be unique across the database. This is problematic because resources for other teams are hidden, so a team...
DAG Gantt chart taskgroup toggle
...their gantt duration as the envelope of the tasks inside it. This makes the gantt chart difficult to interpret, since it makes it possible that some lines on the chart are "very long" even though the actual time TIs spent running might be negligible. # Proposed solution Add a...
Replace `airflow.utils.state import` with `airflow.sdk` in Common AI Provider
## Description > This is an issue that is reserved for the Airflow Summit "Contributors" Workshop. This is denoted with the label `contributors-workshop`. Out of respect for the organizers and participants of this workshop, **please do not implement a PR that addresses this...
Replace `airflow.utils.state import` with `airflow.sdk` in edge3 Provider
## Description > This is an issue that is reserved for the Airflow Summit "Contributors" Workshop. This is denoted with the label `contributors-workshop`. Out of respect for the organizers and participants of this workshop, **please do not implement a PR that addresses this...
Replace `airflow.utils.state` import with `airflow.sdk` in OpenSearch Provider
## Description > This is an issue that is reserved for the Airflow Summit "Contributors" Workshop. This is denoted with the label `contributors-workshop`. Out of respect for the organizers and participants of this workshop, **please do not implement a PR that addresses this...
Airflow Summit Contributors Workshop
## Description This issue is a guide for creating issues and opening PR's for the Contributors Workshop at Airflow Summit. ### Opening PR's for Existing Issues There is currently a set of 25+ Issues that are opened with the label `contributors-workshop` to be used for the...
Replace `airflow.utils.state import` with `airflow.sdk` in Informatica Provider
## Description > This is an issue that is reserved for the Airflow Summit "Contributors" Workshop. This is denoted with the label `contributors-workshop`. Out of respect for the organizers and participants of this workshop, **please do not implement a PR that addresses this...
Add show/hide log level button (Airflow UI)
### Description Right now when we want to copy logs from the task instance we should remove some additional log parts like "INFO - " or timestamp. For timestamp there's a button to hide it. I suggest to add the similar button for hiding log level. ### Use case/motivation It will...
CronDataIntervalTimetable: first run delayed ~2 periods when start_date falls mid-period (catchup=False)
### Description **Correction (2026-09-02): the original version of this issue was wrong about the common case - see below.** I originally claimed `CronDataIntervalTimetable`/`DeltaDataIntervalTimetable` "always wait for the next boundary" on first run under `catchup=False`, by...
Companies and products named
About this problem
Evidence
41 posts from 31 people in 4 places, about 4 a week over 65 days. Mostly on apache/airflow, r/dataengineering, r/mlops. Tools named alongside: GitHub, Claude Code, dbt, SQLite.
Frustration Frustration 0 of 3· Seen on GitHub issues, Reddit
How it was grouped
Posts about Apache Airflow that state a pain but match no known issue yet. First post Sep 1, 2026, latest Nov 4, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method
Other Apache Airflow problems
Apache Airflow is too complex to run well
34 posts from 32 people on GitHub issues.
Apache Airflow jobs fail silently or break often
13 posts from 12 people on GitHub issues, Stack Overflow.
Apache Airflow upgrades break working setups
11 posts from 10 people on GitHub issues.
Debugging Apache Airflow failures is hard
7 posts from 6 people on GitHub issues.
Apache Airflow login, SSO or OAuth breaks
4 posts from 3 people on GitHub issues.
Apache Airflow backfills are slow and risky
4 posts from 4 people on GitHub issues.
Trouble with Apache Airflow retry
Top post: "Task state store writes from `on_retry_callback` fail with 404", one of 3 posts from 3 people on GitHub issues.
Trouble with Apache Airflow dags
Top post: "DagModel.owners falls back to 'airflow' for Dags with only mapped tasks (LazyDeserializedDAG.owner ignores", one of 3 posts from 3 people on GitHub issues.
Trouble with Apache Airflow backend
Top post: "Support [workers] state_store_backend in language SDK tasks", one of 3 posts from 3 people on GitHub issues.
Apache Airflow lock-in makes switching costly
2 posts from 2 people on GitHub issues. Unverified.
Apache Airflow runs out of memory
2 posts from 2 people on GitHub issues. Unverified.
Apache Airflow permissions are confusing and error-prone
2 posts from 2 people on GitHub issues. Unverified.
History