topic Set environment variables in global init scripts in Data Engineering

Set environment variables in global init scripts

Marcel — Fri, 19 Aug 2022 13:53:58 GMT

Hi Databricks Community,

I want to set environment variables for all clusters in my workspace.

The goal is to have environment (dev, prod) specific environment variables values.

Instead of set the environment variables for each cluster, a global script is desired.

I tried different scripts like

but environment variables are not available via

The global init script did not fail.

Any ideas?

Thank you!

Pholo — Sat, 20 Aug 2022 11:07:16 GMT

have you tried the widgets utils from databricks?

dbutils.widgets.text("database", "dev")
 
database_def = dbutils.widgets.getArgument("database")
print(database_def)

You can parametrize it when you run the scripts.

Marcel — Mon, 22 Aug 2022 07:16:21 GMT

The cluster teminated after adding the first line to the global init script.

I think the widget utitliy is only for notebooks.

Doesn't the global init script have to be a valid sh/bash script?

Marcel — Wed, 31 Aug 2022 06:19:44 GMT

Solution:

#!/bin/bash
echo export VARIABLE_NAME=VARIABLE_VALUE >> /etc/environment

The Shebang is required.

The environment variable can then be accessed via (Python):

my_variable = os.environ["VARIABLE_NAME"]

brickster — Fri, 03 Feb 2023 08:51:15 GMT

We have set the env variable at Global Init script as below,

sudo echo DATAENV=DEV >> /etc/environment

and we try to access the variable in notebook that run with "Shared" cluster mode.

import os
print(os.getenv("DATAENV"))

But the env variable is not accessible and getting value as "None".

However, when run the notebook with "No Isolation Shared" cluster, then the env variable is accessed successfully.

Any idea, is there any restriction in accessing environment variables at cluster mode level?