topic Re: How to add environment variable in Data Engineering

How to add environment variable

samalexg — Fri, 04 Sep 2015 04:24:07 GMT

Instead of setting the AWS accessKey and secret Key in hadoopConfiguration, I would like to add those in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

How can I do that in databricks?

Re: How to add environment variable

vida — Fri, 04 Sep 2015 16:33:52 GMT

Hi,

You can create an init script which will run a bash script which can set unix environment variables. Go to Databricks Guide >> AWS Configuration >> Init Scripts for more details.

-Vida

Re: How to add environment variable

samalexg — Mon, 07 Sep 2015 06:05:13 GMT

Thanks for your response. I want to add one more environment variable apart from the AWS properties. I just created the init script as mentioned in documentation. I restarted my cluster. And when I read the env variable like

sys.env.get(envName);

in scala. But it returns nothing. How to check whether my init script is executed or not?

Re: How to add environment variable

vida — Tue, 08 Sep 2015 20:48:45 GMT

Hi,

In the init scipts notebook is details on how to find the output:

Output from all init scripts are generated here: dbfs:/databricks/init/output/

Re: How to add environment variable

samalexg — Wed, 09 Sep 2015 09:19:31 GMT

Yes I've verified that. But nothing is there in dbfs:/databricks/init/output/

Re: How to add environment variable

vida — Wed, 09 Sep 2015 16:43:50 GMT

Did you put an echo command or something in your script so that it will output something?

Re: How to add environment variable

cfregly — Thu, 10 Sep 2015 20:22:20 GMT

@samalexg : are you sure you're writing to the

/etc/environment

file as follows:

otherwise, the env vars are only set for the process that is called to run the script.

i assume you're doing this, but wanted to double check as this has been a common mistake in the past.

dbutils.fs.put(s"""dbfs:/databricks/init/$clusterName/setup-awscreds.sh""","""
#!/bin/bash 
sudo echo AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_HERE >> /etc/environment
sudo echo AWS_SECRET_KEY=YOUR_SECRET_KEY_HERE >> /etc/environment
""", true)

Re: How to add environment variable

samalexg — Fri, 11 Sep 2015 12:03:41 GMT

No. I've just set my env variables in the script. Currently our cluster is in production and I can't test it now. Once I got access I will add some echo and let you know the result

Re: How to add environment variable

samalexg — Fri, 11 Sep 2015 12:04:51 GMT

Yes. I set the env variables as mentioned, but one difference. I've not set it based on cluster. I've added init scripts in dbfs:/databricks/init/ itself? Will that makes the difference?

Re: How to add environment variable

TimChan — Fri, 12 Aug 2016 23:08:50 GMT

I have the same problem. I've added this script and it does not write anything to /etc/environment. The output files from the execution of these scripts are also empty.

Re: How to add environment variable

miklos — Thu, 18 Aug 2016 23:07:16 GMT

Yes, its not supported at the moment. We'll make sure to document that!

Re: How to add environment variable

MikeTrewartha — Thu, 20 Oct 2016 21:28:11 GMT

@Miklos Christine , is writing variables to /etc/environment not supported, or are session/system wide environment variables not supported at all?

Re: How to add environment variable

kruhly — Mon, 27 Feb 2017 21:46:52 GMT

The Init Scripts documentation seem to have moved to: user-guide/advanced/init-scripts.html and are no longer in the AWS Configuration section.

Re: How to add environment variable

jric — Mon, 16 Oct 2017 23:16:15 GMT

It is possible! I was able to confirm that the following post's "Best" answer works: https://forums.databricks.com/questions/11116/how-to-set-an-environment-variable.html

FYI for @Miklos Christine and @Mike Trewartha