feat: Enable configuring redis in docker setup

[c-w]

SUMMARY

This PR adds some configuration options for the docker-compose setup:

1. Enable using a password when connecting to redis
2. Enable using redis as the results backend

These changes are useful in scenarios where we want to use the docker-compose setup together with an existing redis instance and when we potentially want to run the superset UI on a different node from the superset workers without going all the way to using the k8s setup.

TESTING INSTRUCTIONS

For testing the redis password support, apply the following patch:

diff --git a/docker-compose-image-tag.yml b/docker-compose-image-tag.yml
index 9309c6d619..b8c5aad55d 100644
--- a/docker-compose-image-tag.yml
+++ b/docker-compose-image-tag.yml
@@ -38,6 +38,11 @@ services:
     restart: unless-stopped
     volumes:
       - redis:/data
+    entrypoint:
+      - sh
+    command:
+      - -c
+      - printf "requirepass supersecret\n" | exec redis-server -
 
   db:
     env_file:
diff --git a/docker/.env b/docker/.env
index 57575da76e..e5749aa3b3 100644
--- a/docker/.env
+++ b/docker/.env
@@ -49,6 +49,7 @@ POSTGRES_PASSWORD=superset
 PYTHONPATH=/app/pythonpath:/app/docker/pythonpath_dev
 REDIS_HOST=redis
 REDIS_PORT=6379
+REDIS_PASSWORD=supersecret
 
 FLASK_DEBUG=true
 SUPERSET_ENV=development

Now start the stack via TAG=4.0.2 docker compose -f docker-compose-image-tag.yml up and verify that Redis operations work.

For testing the redis results backend support, toggle the feature via echo RESULTS_BACKEND_CACHE_TYPE=RedisCache > docker/.env-local, restart the stack, and verify that Redis operations work.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[hughhhh]

nit: couldn't you just put None inside .getenv call as the default?

REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", None)

[c-w]

This construct is there to cast the empty string to None to disable the password (as cachelib documents password as str | None), but technically it's not necessary as eventually redis-py will cast None back to the empty string anyway (here) so this will do:

Suggested change

	password=REDIS_PASSWORD or None,
	password=REDIS_PASSWORD,

Not applying the change for now pending the discussion about whether or not the PR is something that should be merged in the first place.

[mistercrunch]

These changes are useful in scenarios where we want to use the docker-compose setup together with an existing redis instance

Are you secretly trying to use our docker-compose setup in production!? :)

I'm not against making the dev setup more configurable, but the more we push in that direction, the harder it is to guarantee an simple docker-compose up that just works. Also adding logic/support for new env vars in docker-compose that diverge from what works in production isn't best. Portability of .env-type files and superset_config is good, and having support for a different set of env vars in dev/prod (like this PR does) isn't ideal.

Why not simply improving the docs as to how to configure Redis in any superset_config file (production or dev), and clarifying how to alter docker-compose using docker/superset_config_docker.py? Seems you could simply move the custom code in this PR to a local superset_config and roll with it.

About the idea of exposing more

[c-w]

Are you secretly trying to use our docker-compose setup in production!? :)

Guilty as charged :) I'm not quite at the stage where moving to k8s makes sense but the self-contained nature of container still adds a ton of value so docker-compose has been wonderful.

Why not simply improving the docs as to how to configure Redis in any superset_config file (production or dev), and clarifying how to alter docker-compose using docker/superset_config_docker.py? Seems you could simply move the custom code in this PR to a local superset_config and roll with it.

I hear you. The suggestion definitely works in environments where you have access to a filesystem for setting up a volume mount for the custom config (which isn't always the case, e.g. many cloud offerings that will run your docker image for you won't let you easily do volumes) or it forces you to build your own image that bakes in the custom config (which adds a fair bit of friction on top of being able to just deploy the official image with some environment variable changes). With the changes proposed in this PR, however, the official image turns into something that can be easily configured to externalize all of its state which IMO has value and extends the possible use-cases.

[mistercrunch]

Ah! I've been having this idea to expose more of what's in superset/config.py as env vars in a way that's evolutive / straightforward and is portable from dev/prod.

The idea would be to allow-list at set of keys, and look for matching SUPERSET__ -prefix env var, and do the proper type casting if/where necessary (bool, string, int, ...). Should be pretty straightfoward.

[mistercrunch]

I think I had or someone else had done something similar with Airflow a while back. Reading at what it looks like today -> https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html, there are good ideas in there, like sections and conventions.