cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Marcel
by • New Contributor III
  • 26182 Views
  • 4 replies
  • 2 kudos

Resolved! Set environment variables in global init scripts

Hi Databricks Community,I want to set environment variables for all clusters in my workspace.The goal is to have environment (dev, prod) specific environment variables values.Instead of set the environment variables for each cluster, a global script ...

Data Engineering
Reply
  • 26182 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
brickster
New Contributor II
  • 2 kudos

We have set the env variable at Global Init script as below,sudo echo DATAENV=DEV >> /etc/environmentand we try to access the variable in notebook that run with "Shared" cluster mode. import os print(os.getenv("DATAENV"))But the env variable is not a...

  • 2 kudos
3 More Replies
Anonymous
by • Not applicable
  • 2173 Views
  • 3 replies
  • 5 kudos

What's the recommended way to use secrets within Global init scripts?

Data Engineering
Reply
  • 2173 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
sstrong99
New Contributor II
  • 5 kudos

I also have this question, and I don't understand the answer from @Sajith Appukuttan​ . This solution would require configuring each new cluster separately with the correct environment variable, correct? That seems like it defeats the purpose of usi...

  • 5 kudos
2 More Replies
isaac_gritz
by • Databricks Employee
  • 3265 Views
  • 6 replies
  • 8 kudos

Library Dependency

How to Install Libraries on DatabricksYou can install libraries in Databricks at the cluster level for libraries commonly used on a cluster, at the notebook-level using %pip, or using global init scripts when you have libraries that should be install...

Data Engineering
Reply
  • 3265 Views
  • 6 replies
  • 8 kudos
View Replies
Latest Reply
Chris_Shehu
Valued Contributor III
  • 8 kudos

It can be a risky to install libraries without any sort of oversite/security structure to ensure those libraries have no vulnerabilities. I think more caution needs to be added to the wording of these documents to express that. All of the libraries w...

  • 8 kudos
5 More Replies
Labels