Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deferred / deferrable task are not take in account by max_active_tasks #40528

Open
1 of 2 tasks
raphaelauv opened this issue Jul 1, 2024 · 0 comments
Open
1 of 2 tasks
Labels
affected_version:2.9 area:async-operators AIP-40: Deferrable ("Async") Operators area:Triggerer kind:bug This is a clearly a bug

Comments

@raphaelauv
Copy link
Contributor

raphaelauv commented Jul 1, 2024

Apache Airflow version

2.9.2

What happened?

If you set max_active_tasks, it do not limit the number of concurrent deferred tasks.

It's possible to restrict the number of concurrent deferred tasks with a pool but I want limit the number of tasks run by a dag and keep the possibility for every task to use a specific pool ( we can only use 1 pool by task in airflow )

What you think should happen instead?

max_active_tasks should take in account the deferred task or we should create a new boolean setting

max_active_tasks_include_deferred like what was done for the pools

How to reproduce

from datetime import datetime, timezone, timedelta

from airflow.sensors.time_sensor import TimeSensorAsync
from pendulum import today
from airflow import DAG

with DAG(
        dag_id='example',
        schedule_interval='0 0 * * *',
        max_active_tasks=10,
        start_date=today("UTC").add(days=-1)):
    for i in range(100):
        TimeSensorAsync(task_id=f"a_{i}", target_time=(datetime.now(tz=timezone.utc) + timedelta(minutes=10)).time())

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@raphaelauv raphaelauv added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Jul 1, 2024
@dosubot dosubot bot added the area:async-operators AIP-40: Deferrable ("Async") Operators label Jul 1, 2024
@RNHTTR RNHTTR added area:Triggerer and removed area:core needs-triage label for new issues that we didn't triage yet labels Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:2.9 area:async-operators AIP-40: Deferrable ("Async") Operators area:Triggerer kind:bug This is a clearly a bug
Projects
None yet
Development

No branches or pull requests

3 participants