Hello,
I would like to install some libraries (both public and private) on a job cluster. I am using Azure Data Factory to run my Databricks notebooks and hence would like to use job clusters to run these jobs.
I have passed my init script to the job cluster but sometimes the package installs work, sometimes not, with no real pattern. The workspace paths to my packages well exist and are correctly set up.
What's wrong ? Is there anything I should check ? Is there another more robust way to do it so that it always works? Since it's not robust, sometimes my libraries are well installed on my job cluster, sometimes not.
I have attached the configuration I am using in Azure Data Factory to use my init script, and also a screenshot of what my init script looks like.
adf config for init script
init script
Thank you very much in advance for the help,
Sacha