You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We need to use BranchPythonOperator and dynamic task mapping in a DAG to meet our requirements. The BranchPythonOperator controls whether dynamic mapping is needed, and there is a task (branch_2_task) following the dynamic task mapping that must be executed regardless of whether the dynamic task mapping runs (it can be skipped by receiving an empty list). As shown in the diagram 1, the trigger_rule for branch_2_task needs to be set to none_failed.
diagram 1:
If it is set to all_success, the task branch_2_task will also be skipped if the dynamic mapping is skipped due to an empty list (diagram 2).
diagram 2:
However, when BranchPythonOperator triggers the branch_1 task, the task branch_2_task on the other branch also gets executed (diagram 3).
diagram 3:
The potential bug could be that Airflow does not check if a task is on the executing branch but only checks the trigger_rule.
Could you fix this?
What you think should happen instead?
The task branch_2_task has trigger_rule none_failed to be executed even if the dynamic task mapping before it is skipped. If the other branch is executed, the task branch_2_task must be skipped because the branch should be skipped.
In the third picture the branch_2_task trigger rule set to none_failed,since all the previous tasks are not failed (skipped or success) it should run.
the problem is actually, in the 3rd picture the task branch_2_task should not run, because the branch operator decided to trigger branch_1 (the other branch).
I see.
I still don't know if it's a bug though because the reason I mention, so we will wait for third opinion.
A quick action that I think of to avoid it, in branch_2_task you can retrive the task_id from the branch_operator and decide to run the logic or not by this.
Apache Airflow version
2.9.2
If "Other Airflow 2 version" selected, which one?
No response
What happened?
We need to use BranchPythonOperator and dynamic task mapping in a DAG to meet our requirements. The BranchPythonOperator controls whether dynamic mapping is needed, and there is a task (branch_2_task) following the dynamic task mapping that must be executed regardless of whether the dynamic task mapping runs (it can be skipped by receiving an empty list). As shown in the diagram 1, the trigger_rule for branch_2_task needs to be set to none_failed.
diagram 1:
![image](https://private-user-images.githubusercontent.com/2765326/346110818-5d12e4d8-365a-40ef-b554-09a5729fecbc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjExNTcyNDUsIm5iZiI6MTcyMTE1Njk0NSwicGF0aCI6Ii8yNzY1MzI2LzM0NjExMDgxOC01ZDEyZTRkOC0zNjVhLTQwZWYtYjU1NC0wOWE1NzI5ZmVjYmMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTZUMTkwOTA1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZjA2OTM3ODE0MGRhM2Y1M2Y5N2UwMDhkNmRhMDgzZmQ5OGIxNTYxYWNkMTQ3ZjBjOWNmMmYwMGY3ZDdjYmEzOCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.cG9E379e-sqHDnZ0lBviiaFx_Ql2bfdrVGGqWGGTkC4)
If it is set to all_success, the task branch_2_task will also be skipped if the dynamic mapping is skipped due to an empty list (diagram 2).
diagram 2:
![image](https://private-user-images.githubusercontent.com/2765326/346111160-f063a815-2976-44d8-8006-fa80ebaae687.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjExNTcyNDUsIm5iZiI6MTcyMTE1Njk0NSwicGF0aCI6Ii8yNzY1MzI2LzM0NjExMTE2MC1mMDYzYTgxNS0yOTc2LTQ0ZDgtODAwNi1mYTgwZWJhYWU2ODcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTZUMTkwOTA1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZGNlYWViMDkxY2U4ZWY3NjA1NmZlYzViZWQxNTdjMjhiZWNhMjYyOTE4NjE0YmE2NjJjZmM4ZWY0MDBmNDlhNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.O42HcHhMzQEIP8ypWIAJ9MCLPoPyQCj1vunMwtMK6zE)
However, when BranchPythonOperator triggers the branch_1 task, the task branch_2_task on the other branch also gets executed (diagram 3).
diagram 3:
![image](https://private-user-images.githubusercontent.com/2765326/346111505-31a2c2e6-112f-42c5-b8d0-4911bed161f7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjExNTcyNDUsIm5iZiI6MTcyMTE1Njk0NSwicGF0aCI6Ii8yNzY1MzI2LzM0NjExMTUwNS0zMWEyYzJlNi0xMTJmLTQyYzUtYjhkMC00OTExYmVkMTYxZjcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTZUMTkwOTA1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NjE1MmUxMGQzOTBlNGIwNTQ1MGM1MzhiNGM2ZGQ1YWNmZGM5YjMwNmMwNmQ3NjA3YzAyMjhmNjExMjRmMzRjMSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.bjHELuo3CDY11PGhWebfzZpTjgOOov7OcMXTf1lOLRE)
The potential bug could be that Airflow does not check if a task is on the executing branch but only checks the trigger_rule.
Could you fix this?
What you think should happen instead?
The task branch_2_task has trigger_rule none_failed to be executed even if the dynamic task mapping before it is skipped. If the other branch is executed, the task branch_2_task must be skipped because the branch should be skipped.
How to reproduce
Operating System
Ubuntu
Versions of Apache Airflow Providers
No response
Deployment
Other
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: