topic Are their plans to support init scripts within shared compute resources? in Data Governance

Are their plans to support init scripts within shared compute resources?

Chris_Shehu — Wed, 22 Mar 2023 22:13:21 GMT

Currently we have an Init script that specifies the location of different network resources by updating the hosts file. When we moved to the Unity catalog all of our clusters had to be upgraded to support it. Now our scripts have stopped working due to TCP/IP failures. I couldn't find the answer easily and the only reference I've been able to find is a forum post by @Hubert Dudek OracleDB shared Cluster / DBFS on Unity Catalog Cluster (databricks.com) showing that init scripts won't work on shared clusters. Are there any plans to change this functionality?

Re: Are their plans to support init scripts within shared compute resources?

Anonymous — Thu, 23 Mar 2023 04:11:29 GMT

@Christopher Shehu :

At the moment, shared compute resources in Databricks do not support init scripts. This is because shared compute resources are intended for lightweight, ephemeral workloads and do not provide the same level of control over the underlying infrastructure as dedicated clusters.

However, Databricks is continuously improving its platform and adding new features, so it's possible that support for init scripts on shared compute resources could be added in the future. In the meantime, you may need to consider alternative approaches for setting up network resources, such as using a custom Docker container or configuring the resources manually in your notebooks.

With Unity Catalog - shared compute resources in Databricks do support init scripts

Re: Are their plans to support init scripts within shared compute resources?

Chris_Shehu — Thu, 23 Mar 2023 08:18:49 GMT

Good morning I'm a little confused. Initially you say that shared compute resources do not support init scripts but then at the end you say they do? How does this work? We're using unity catalog and it doesn't appear to.

As far as improvements I would suggest adding some sort of indicator in the compute resource screen to clarify what is and isn't available. I spent hours troubleshooting why my code was failing and only discovered the reason after looking at the compute logs showing that the init scripts aren't' running.

Re: Are their plans to support init scripts within shared compute resources?

Hemant — Sun, 26 Mar 2023 17:19:35 GMT

Hi @Christopher Shehu, did you try the global script in place of the init script,?

Re: Are their plans to support init scripts within shared compute resources?

youssefmrini — Wed, 29 Mar 2023 10:01:31 GMT

We are bringing soon init script and Scala to shared Cluster

Re: Are their plans to support init scripts within shared compute resources?

Chris_Shehu — Wed, 29 Mar 2023 12:12:08 GMT

Thanks @Youssef Mrini !