Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-17-2023 02:28 PM
Hi @Sulfikkar Basheer Shylaja , Why don't you store the init-script on DBFS and just pass the dbfs:/ path of the init script in Pulumi? You could just run this code on a notebook-
%python
dbutils.fs.put("/databricks/init-scripts/set-private-pip-repositories-relay-feed-init-script.sh", """
#!/bin/bash
if [-n $DEVOPS_GIT_PAT]; then
use $DEVOPS_GIT_PAT
else
echo "Couldn't fetch PAT token"
fi
printf "[global]\n" > /etc/pip.conf
printf "extra-index-url=" >> /etc/pip.conf
printf "https://$DEVOPS_GIT_PAT@artificat_url/pypi/simple/\n" >> /etc/pip.conf
echo "script executed"
""", True)This is using dbutils command to upload the script to DBFS in the location dbfs:/databricks/init-scripts/set-private-pip-repositories-relay-feed-init-script.sh
For using Databricks secrets scope, you are correct. Assign the secret to a spark environment variable and reference it in the init script. Please follow these two docs -
https://www.pulumi.com/registry/packages/databricks/api-docs/cluster/#state_spark_env_vars_python
Let me know if this info helps you.