@samalexg​ : are you sure you're writing to the

/etc/environment

file as follows:

otherwise, the env vars are only set for the process that is called to run the script.

i assume you're doing this, but wanted to double check as this has been a common mistake in the past.

dbutils.fs.put(s"""dbfs:/databricks/init/$clusterName/setup-awscreds.sh""","""
#!/bin/bash 
sudo echo AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_HERE >> /etc/environment
sudo echo AWS_SECRET_KEY=YOUR_SECRET_KEY_HERE >> /etc/environment
""", true)