- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2025 03:07 AM
Hi All,
According the documentation of delta live tables (https://docs.databricks.com/en/delta-live-tables/external-dependencies.html), one can user either global or cluster scoped init scripts.
However, I don't see an option to select init scripts in the DLT UI? Do I need to set this as a (key,value ) Configuration under the advanced tab? We need this script to set our pip configuration correctly (otherwise we can not install any python package due to networking issues)
Is there an example (YAML/JSON) of a DLT that uses either a global or cluster-scoped init script?
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2025 04:44 AM
Hi,
You can do it via a cluster policy. It can be achieved in two steps:
- create a cluster policy with required attributes.You can provide the init_scripts attribute in the policy. For reference: https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-definition#attributes
- Once the policy is created then choose this policy under cluster settings of DLT pipeline. For reference https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/configure-compute#cluster-polic...
