Thanks @Lukasz Lu​ - that worked for me as well. When I used the following script:

#!/bin/bash
echo MY_TEST_VAR=value1 | tee -a /etc/environment >> /databricks/spark/conf/spark-env.sh

• for non-docker clusters, MY_TEST_VAR shows up twice in ` /databricks/spark/conf/spark-env.sh`
• for docker clusters MY_TEST_VAR shows up once
• in both cases `os.getenv("MY_TEST_VAR")` gives value1 as expected