Env_vars from KubernetesPodOperator does not assume secret (deploy as env var) because secrets are loaded after env vars into the pods.

[ana-carolina-januario]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.6.0

What happened?

We've tried to perform some bash commands where we need to use env_vars to build the command.
The main environment variable, so called COMMAND, has a placeholder to be replaced by some secret values, for instance:
(here having (), {}, or nothing surrounding the env_var, does not affect the result.)

COMMAND="echo 'This is my secret $(SECRET_AS_ENV_VAR)'"

but when I run:
echo $COMMAND

The result does not replace the variable.

When I run:
echo $SECRET_AS_ENV_VAR
The value is printed correctly so, the variable is successfully loaded.

I've tried to

I've analyzed the pods and noticed the order env_vars are presented, might be affecting this replacement.

I've notice that when the pod is being build, the secrets (as env vars) are being loaded after the env vars. This could be reviewed in order to load secrets prior to the load of env_vars.

Thanks,
Ana

What you think should happen instead?

I believe the env var COMMAND should have the SECRET_AS_ENV_VAR replaced properly when starting the pod.

After looking at the code, I've noticed this function should load secrets at the same time (or even before) loading env vars.:
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/7.3.0/_modules/airflow/providers/cncf/kubernetes/operators/pod.html#KubernetesPodOperator.build_pod_request_obj

How to reproduce

Using your KubernetesPodOperator example,
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/1.2.0/_modules/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.html
add a variable with a value that uses the secret deployed as env:

init_environments = [k8s.V1EnvVar(name='key1', value='This string should show the value of SQL_CONN = $(SQL_CONN)'), k8s.V1EnvVar(name='key2', value='value2')]

For the args parameter:
args=['echo $key1']

With this echo, the value won't be replaced in the value of key1.
But if you run 'echo $SQL_CONN' the value will be correctly printed.

Operating System

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian

Versions of Apache Airflow Providers

Version of the apache-airflow-providers-cncf-kubernetes is 6.1.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ana-carolina-januario]

Hi,
Additionally, I've tried multiple workarounds:

1. export to other env var like:
   export COMMAND_1=$COMMAND
   without success.
2. export the secret to another env var(using the cmds parameter) and then export the command to another env_var as well:
   cmds=['sh', '-c', 'export NEW_SECRET=$(SQL_CONN) ; export COMMAND_1=$(COMMAND)']
   also without changes on the behaviour.

Thanks,
Ana.

[raphaelauv]

use kubernetes native secret ( since kubernetes env_vars are plain text )

secret_toto = Secret(
    deploy_type="env",
    deploy_target="THE_SECRET",
    secret="toto",
    key="tata")

KubernetesPodOperator(
    task_id="task-toto",
    kubernetes_conn_id="kubernetes_default",
    image="aaaaa:0.1",
    secrets=[secret_toto],
)

[ana-carolina-januario]

I'm not sure if I understood the suggestion.
My problem is that, once the pod is launched, the order of the env_vars (secrets+env_vars) is not the ideal. - if they are independent fro each other, the order does not matter.

I would expect the secrets being loaded before the env_vars but once I run "describe" from kubectl for the pod, I figure that the secrets are loaded after the env_vars (that I build dynamically and hence it is not compatible to put them in a secret).
Once I launch the pod, I 'need' the secrets to be loaded before the env vars so I can perform all actions using env_vars that refer secrets' values.

Let me know if this does not make sense to you.

Thanks!

[potiuk]

This is the way how env vars are passed - there is no way you can add secrets to env vars before - because they are evaluated in two different places. When you create k8s pod, you pass it the env vars and K8S processes the pod definition and sets the env vars. Only then it launches the POD and airflow process start and it's the airflow process that reads secrets. If you do it the other way - this is also a terrible, terrible security fix - as @raphaelauv mentioned - env variables are visible and quereable when you have access to K8S. so you should absoiutely like NEVER pass the env variables with resolved secrets to launch a pod,

You need to rethink your strategty. What @raphaelauv suggest is good - make your secrets available as K8S secrets, and in your PODs you can mount the secrets as env variables (if you can make your secrets available as K8S secrets). If you need to retrieve them dynamically by Airflow from secrets manager, you need to write a code that will do it dynamically as part of your task execute() mehod or JINJA template that is resolved after the pod has been instantiated.

[ephraimbuddy]

Closing as won't fix

[potiuk]

Also we just merged #40519 that addresses that by explicitly mentioning in appropriate places that env variables are bad way of passing secrets.

[ana-carolina-januario]

Actually I am using k8s native secrets to pass credentials/secrets.
What I pass as env var is a command with the reference for a secret deployed as env_Var:
from airflow.kubernetes.secret import Secret
secret_value = Secret(
deploy_type=deploy_type,
deploy_target=secret_key,
secret=secret_name,
key=value
)
and my command is like:
COMMAND="some-tool $(SECRET_DEPLOYED_AS_ENV_VAR)"
where some-tool needs the value passed through the env_var that contains the secret value.

I am not passing the value of the secret directly as an env var, I am passing through k8s secret, deploying as env_var.

Thanks!

[potiuk]

I am not passing the value of the secret directly as an env var, I am passing through k8s secret, deploying as env_var.

I stull do not understand. If you use native secret in K8S you should be able to mount it as ENV var directly - and value of that secret should be available for you - this is native k8s behaviour, and Airflow has nothing to do with it.

I really have no idea in this case what is passed when and what problem you have - maybe I am stupid, but most likely You should explain in ore detail what your problem is.

Please try to explain it in the way that we can understand it if you need help.

I will re-open it and convert into discussion so that you can continue explaining it, but this is almost for sure not an airflow issue, so discussion is more appropriate.

[potiuk]

The best is if you just copy&paste your exact DAG pieces with the bash command and explain it with examples. I believe you should also read more on how you can use k8s secrets and mount them as variables - this is more than likely to solve your problems without even discussing airflow and doing some fancy processing with variables and bash commands.

[ana-carolina-januario]

Sure!
We are running spark jobs using KubernetesporOperator.
We have 2 use-cases, one works and the other doesn't:
One, we use secrets and send the command directly through KubernetesPodOperator. Here we do not have any issue since we have all env vars and when the command is executed, the env_vars used in the command are correclty replaced.
The second case we need to manupilate the command in the image that we use and we are passing the command as an env_var.
The partial command is:
COMMAND="spark-submit --name Our_Job_Name
--master k8s://https://some.link.com
--class ourClass.SparkQueryExecutor
--driver-cores 1
--driver-memory 1g
--executor-cores 1
--executor-memory 1g
--num-executors 1
--deploy-mode cluster
--conf spark.hadoop.fs.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.access.key=${AWS_ACCESS_KEY_ID}
--conf spark.hadoop.fs.s3a.secret.key=${AWS_SECRET_ACCESS_KEY}
--conf spark.hadoop.fs.s3a.fast.upload=true
--conf spark.kubernetes.node.selector.ulysses/node_name=jobs
--conf spark.kubernetes.driver.podTemplateFile=s3a://some_bucket/some_object "

The secrets are used the same way.

When I was comparing the exact command running with the 2 different ways, I noticed that, the order how Airflow loads the environment variables matters and it loads the env_vars first, and the secrets are loaded after. With this way, my approach will never work.

This is the line of your code where I found the secrets being loaded as env vars:

airflow/airflow/providers/cncf/kubernetes/operators/pod.py

Line 807 in eb24742

for secret in self.secrets:

since the env_vars are loaded first, here:

airflow/airflow/providers/cncf/kubernetes/operators/pod.py

Line 773 in eb24742

env=self.env_vars,

This is the output of running 'kubectl describe pod name_of_our_pod':

here we can see the secrets are defined after my env_vars.

I've tried to export the command to a new env var by running 'export COMMAND_1=$COMMAND' but it did not work.

Here it is a small sample on the example that does not work.

Let me know if you need more info.

Thanks!!

[potiuk]

No. This is misunderstanding how command works and it has nothing to do with Airflow. if you are sending a command to execute and you want part of it to read values from environment variables you need to have something to replace the values - i.e. interpolate the variables. In your case your command has ${AWS...} specified, but they are just strings because nothing interpolates the variables.

What you really want to do (if you want to do it this way) you need to wrap your command in a bash interpreter, so that the bash interpreter can interpolate the variables, but this will be quite spaghetti of code and quotes, which you need to figure out.

You should experiment with something simple and learn from it. Smth like this (but I am not sure if this is right - you need to figure it out):

When you do this:

COMMAND=`echo ${AWS_ACCESS_KEY_ID}'

The command will just output literally ${AWS...}. But when you do this:

COMMAND=`bash -c "echo ${AWS_ACCESS_KEY_ID}"'

It should do what you want to do. This will work in the way that it will run bash interpreter, that will launch the command specified via -c - this way the bash interpreter will be able to interpolate the variables you refer to and replace them with their actual values taken from the environment variables.