Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ObjectStoragePath should be template-able #40361

Open
1 of 2 tasks
fritz-astronomer opened this issue Jun 21, 2024 · 2 comments · May be fixed by #40638
Open
1 of 2 tasks

ObjectStoragePath should be template-able #40361

fritz-astronomer opened this issue Jun 21, 2024 · 2 comments · May be fixed by #40638
Labels
area:core good first issue kind:feature Feature Requests

Comments

@fritz-astronomer
Copy link
Contributor

Description

ObjectStoragePath should be able to handle Jinja Templates

Use case/motivation

If I have an operator, that has an ObjectStoragePath argument

class MyOperator(BaseOperator):
  template_fields = ('path',)

  def __init__(path, **kwargs):
    self.path = path
    super().__init__(**kwargs)

and am using it in a DAG, with templating

MyOperator(
  ...,
  path=ObjectStoragePath(
    "s3://my_bucket/foo/{{ ds_nodash }}/file.csv",
    conn_id="s3"
  )
)

Templating didn't seem to work

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@fritz-astronomer fritz-astronomer added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Jun 21, 2024
@fritz-astronomer
Copy link
Contributor Author

acc'd to this https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating
it looks like ObjectStoragePath doesn't have template_fields =

I tried a few things:

ObjectStoragePath.template_fields = (...)
doesn't work, because the init field is *args (I tried with args)

I can't do render_template(...) in my operator, I think, because ObjectStoragePath is read-only

@shahar1 shahar1 added area:core and removed needs-triage label for new issues that we didn't triage yet labels Jun 27, 2024
@gyli
Copy link

gyli commented Jul 7, 2024 •
edited
Loading

The current render_template does not handle the case when value is ObjectStoragePath, and I think it can be added.

As a workaround before it's added, I think you can override render_template in your operator to render and rebuild the path.

@gyli gyli linked a pull request Jul 7, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:core good first issue kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: support rendering ObjectStoragePath value gyli/airflow
4 participants
@potiuk @gyli @shahar1 @fritz-astronomer