Vivian_Wilfred
Databricks Employee
Databricks Employee

Hi @Sulfikkar Basheer Shylaja​ , Why don't you store the init-script on DBFS and just pass the dbfs:/ path of the init script in Pulumi? You could just run this code on a notebook-

%python
dbutils.fs.put("/databricks/init-scripts/set-private-pip-repositories-relay-feed-init-script.sh", """
#!/bin/bash
if [-n  $DEVOPS_GIT_PAT]; then
    use $DEVOPS_GIT_PAT
else
    echo "Couldn't fetch PAT token"
fi
printf "[global]\n" > /etc/pip.conf
printf "extra-index-url=" >> /etc/pip.conf
printf "https://$DEVOPS_GIT_PAT@artificat_url/pypi/simple/\n" >> /etc/pip.conf
echo "script executed"
""", True)

This is using dbutils command to upload the script to DBFS in the location dbfs:/databricks/init-scripts/set-private-pip-repositories-relay-feed-init-script.sh

For using Databricks secrets scope, you are correct. Assign the secret to a spark environment variable and reference it in the init script. Please follow these two docs -

https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secrets#reference-a-secret-in-an...

https://www.pulumi.com/registry/packages/databricks/api-docs/cluster/#state_spark_env_vars_python

Let me know if this info helps you.