How to add environment variable
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-03-2015 09:24 PM
Instead of setting the AWS accessKey and secret Key in hadoopConfiguration, I would like to add those in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
How can I do that in databricks?
- Labels:
-
AWS
-
Aws account
-
Notebooks
-
Variables
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2015 09:33 AM
Hi,
You can create an init script which will run a bash script which can set unix environment variables. Go to Databricks Guide >> AWS Configuration >> Init Scripts for more details.
-Vida
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2015 11:05 PM
Thanks for your response. I want to add one more environment variable apart from the AWS properties. I just created the init script as mentioned in documentation. I restarted my cluster. And when I read the env variable like
sys.env.get(envName);
in scala. But it returns nothing. How to check whether my init script is executed or not?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-27-2017 01:46 PM
The Init Scripts documentation seem to have moved to: user-guide/advanced/init-scripts.html and are no longer in the AWS Configuration section.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-08-2015 01:48 PM
Hi,
In the init scipts notebook is details on how to find the output:
- Output from all init scripts are generated here: dbfs:/databricks/init/output/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-09-2015 02:19 AM
Yes I've verified that. But nothing is there in dbfs:/databricks/init/output/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-09-2015 09:43 AM
Did you put an echo command or something in your script so that it will output something?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-11-2015 05:03 AM
No. I've just set my env variables in the script. Currently our cluster is in production and I can't test it now. Once I got access I will add some echo and let you know the result
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-10-2015 01:22 PM
@samalexg : are you sure you're writing to the
/etc/environment
file as follows:
otherwise, the env vars are only set for the process that is called to run the script.
i assume you're doing this, but wanted to double check as this has been a common mistake in the past.
dbutils.fs.put(s"""dbfs:/databricks/init/$clusterName/setup-awscreds.sh""","""
#!/bin/bash
sudo echo AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_HERE >> /etc/environment
sudo echo AWS_SECRET_KEY=YOUR_SECRET_KEY_HERE >> /etc/environment
""", true)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-11-2015 05:04 AM
Yes. I set the env variables as mentioned, but one difference. I've not set it based on cluster. I've added init scripts in dbfs:/databricks/init/ itself? Will that makes the difference?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2016 04:08 PM
I have the same problem. I've added this script and it does not write anything to /etc/environment. The output files from the execution of these scripts are also empty.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-18-2016 04:07 PM
Yes, its not supported at the moment. We'll make sure to document that!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2016 02:28 PM
@Miklos Christine , is writing variables to /etc/environment not supported, or are session/system wide environment variables not supported at all?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2017 04:16 PM
It is possible! I was able to confirm that the following post's "Best" answer works: https://forums.databricks.com/questions/11116/how-to-set-an-environment-variable.html
FYI for @Miklos Christine and @Mike Trewartha