Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deferred tasks get kill during heartbeat callback in some rare cases #40435

Open
2 tasks done
0x26res opened this issue Jun 26, 2024 · 0 comments · May be fixed by #40453
Open
2 tasks done

deferred tasks get kill during heartbeat callback in some rare cases #40435

0x26res opened this issue Jun 26, 2024 · 0 comments · May be fixed by #40453
Assignees
@0x26res
Labels
affected_version:2.8 Issues Reported for 2.8 area:Scheduler Scheduler or dag parsing Issues good first issue kind:bug This is a clearly a bug

Comments

@0x26res
Copy link
Contributor

0x26res commented Jun 26, 2024

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.8.1 (MWAA)

What happened?

My airflow deferred (AWS) BatchOperator occasionally fail. When this happens I can see this:

{{local_task_job_runner.py:302}} WARNING - State of this instance has been externally set to deferred. Terminating instance.

I actually think it's an oversight in the code for local_task_job_runner. This line should check for state being RUNNING or DEFERRED.

What you think should happen instead?

No response

How to reproduce

This is hard to reproduce. The issue is very transient.

Operating System

MWAA

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.16.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

MWAA

Anything else?

I would say it happens one out of 100 run.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@0x26res 0x26res added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Jun 26, 2024
@0x26res 0x26res linked a pull request Jun 27, 2024 that will close this issue
Open
@shahar1 shahar1 added area:Scheduler Scheduler or dag parsing Issues and removed needs-triage label for new issues that we didn't triage yet area:core labels Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@0x26res 0x26res

Labels
affected_version:2.8 Issues Reported for 2.8 area:Scheduler Scheduler or dag parsing Issues good first issue kind:bug This is a clearly a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Consider deferred tasks as running in heartbeat 0x26res/airflow
3 participants
@0x26res @eladkal @shahar1