topic Re: Delta Live Tables - use cluster-scoped init scripts in Data Engineering

Delta Live Tables - use cluster-scoped init scripts

MichielPovre — Wed, 22 Jan 2025 11:07:19 GMT

Hi All,

According the documentation of delta live tables (https://docs.databricks.com/en/delta-live-tables/external-dependencies.html), one can user either global or cluster scoped init scripts.

However, I don't see an option to select init scripts in the DLT UI? Do I need to set this as a (key,value ) Configuration under the advanced tab? We need this script to set our pip configuration correctly (otherwise we can not install any python package due to networking issues)

Is there an example (YAML/JSON) of a DLT that uses either a global or cluster-scoped init script?

Re: Delta Live Tables - use cluster-scoped init scripts

AngadSingh — Wed, 22 Jan 2025 12:44:47 GMT

Hi,

You can do it via a cluster policy. It can be achieved in two steps:

create a cluster policy with required attributes.You can provide the init_scripts attribute in the policy. For reference: https://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-definition#attributes
Once the policy is created then choose this policy under cluster settings of DLT pipeline. For reference https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/configure-compute#cluster-policy