I run some jobs in the Databricks environment where some resources need authentication. I do this (and I need to) through the vault-cli in the init-script.
However, every time in the init-script I need to install vault-cli and other libraries. Is there any way I can have them pre-installed somehow? I would like to avoid this installation every time I run a job
@João Victor Albuquerque :
Yes, there are a few ways to pre-install libraries and tools in the Databricks environment:
You can choose the approach that best fits your needs and preferences.
I currently use this first option (init-scripts). But my intention is not to need to be installing every time a cluster starts. I wanted one with the libraries already installed in the environment. It seems to me that the 2nd and 3rd option would have that. Is there any documentation for them? Especially the second option. I didn't find anything about it
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!