Data and AI struggles

1,730 problems · 342 companies

Data engineering

Debugging orchestration and DAG failures is hard

5 posts from 5 people5d activeFading, +0 this week

Posts per day

Posts per day5 posts, Aug 31 to Nov 9

The posts behind it

GitHub issuesapache/airflow · hanxdatadogOct 30

Feature: Bind Dag tags into task log context natively, so every task log line carries them

...and filtering (e.g. "send this team its failed-task logs") can't use Dag tags. The workaround today is a listener that calls `structlog.contextvars.bind_contextvars(...)` in `on_task_instance_running`. We run this in production and it mostly works, but it has two gaps that can't...

1 replies
GitHub issuescrewAIInc/crewAI · EndlessRoad-maxOct 24

[BUG] Skipped ConditionalTask passes previous-run output to downstream context

### Description When the same `Crew` instance runs twice, a `ConditionalTask` that executes in the first run and is skipped in the second retains its previous `task.output`. A downstream task with explicit `context=[conditional_task]` then receives that previous output in its...

1 replies
GitHub issuesPrefectHQ/prefect · jasperyueOct 8

Runner SIGTERM handler leaves astop unawaited and waits for served flow subprocesses to finish

### Bug summary Sending SIGTERM to the parent process running `flow.serve()` emits `RuntimeWarning: coroutine 'Runner.astop' was never awaited`. The parent then stays alive until the running flow finishes naturally. The flow is not cancelled. Reproduced twice with Prefect 3.8.7...

GitHub issuesapache/iceberg · HenryCaiHaiyingSep 8

Proposal: KAFKA CONNECT: JMX metrics for the iceberg-kafka-connect commit pipeline

...task happens to hold the coordinator role. There is no time series to alert on, and no way to see whether commits are getting slower before they start failing outright. 2. **Partial vs. full commits.** A timeout-driven partial commit (`iceberg.control.commit.timeout-ms`) means...

GitHub issuesPrefectHQ/prefect · vyagubovAug 31

`dbt source freshness` logs are not associated with the enclosing task

### Bug summary When calling `PrefectDbtRunner.invoke(['source', 'freshness', ...])` inside a Prefect task, logs from source node events appear in the **flow run** only - not in the task that called `invoke()`. ### Steps to reproduce Source freshness logs are missing from the...

3 replies

Companies and products named

2 posts
2 posts
2 posts
1 post
About this problem

Evidence

5 posts from 5 people in 4 places, about 1 a week over 61 days. Mostly on PrefectHQ/prefect, apache/iceberg, crewAIInc/crewAI. Tools named alongside: Prefect, GitHub, Observe, dbt.

Frustration Frustration 2 of 3· Seen on GitHub issues

How it was grouped

Posts that state a pain and match the "orchestration-debugging" rule. First post Aug 31, 2026, latest Oct 30, 2026. Corroborated: 3 or more posts from 2 or more people or places. Method

The same problem elsewhere

Debugging Apache Airflow failures is hard

7 posts from 6 people on GitHub issues.

History

Momentum went from steady to fading.

Nov 6, 2026

Momentum went from fading to steady.

Nov 4, 2026

Momentum went from steady to fading.

Oct 31, 2026

Momentum went from fading to steady.

Oct 29, 2026

Momentum went from steady to fading.

Oct 15, 2026

Corroborated: 3 or more posts from 2 or more people or places.

Oct 14, 2026

Momentum went from fading to steady.

Oct 14, 2026

First seen with 2 posts.

Sep 25, 2026