cancel
Showing results for 
Search instead for 
Did you mean: 

Are their plans to support init scripts within shared compute resources?

Chris_Shehu
Valued Contributor III

Currently we have an Init script that specifies the location of different network resources by updating the hosts file. When we moved to the Unity catalog all of our clusters had to be upgraded to support it. Now our scripts have stopped working due to TCP/IP failures. I couldn't find the answer easily and the only reference I've been able to find is a forum post by @Hubert Dudek​ OracleDB shared Cluster / DBFS on Unity Catalog Cluster (databricks.com) showing that init scripts won't work on shared clusters. Are there any plans to change this functionality?

1 ACCEPTED SOLUTION

Accepted Solutions

youssefmrini
Honored Contributor III
Honored Contributor III

We are bringing soon init script and Scala to shared Cluster

View solution in original post

5 REPLIES 5

Anonymous
Not applicable

@Christopher Shehu​ :

At the moment, shared compute resources in Databricks do not support init scripts. This is because shared compute resources are intended for lightweight, ephemeral workloads and do not provide the same level of control over the underlying infrastructure as dedicated clusters.

However, Databricks is continuously improving its platform and adding new features, so it's possible that support for init scripts on shared compute resources could be added in the future. In the meantime, you may need to consider alternative approaches for setting up network resources, such as using a custom Docker container or configuring the resources manually in your notebooks.

With Unity Catalog - shared compute resources in Databricks do support init scripts

Chris_Shehu
Valued Contributor III

Good morning I'm a little confused. Initially you say that shared compute resources do not support init scripts but then at the end you say they do? How does this work? We're using unity catalog and it doesn't appear to.

As far as improvements I would suggest adding some sort of indicator in the compute resource screen to clarify what is and isn't available. I spent hours troubleshooting why my code was failing and only discovered the reason after looking at the compute logs showing that the init scripts aren't' running.

Hemant
Valued Contributor II

Hi @Christopher Shehu​, did you try the global script in place of the init script,?

Hemant Soni

youssefmrini
Honored Contributor III
Honored Contributor III

We are bringing soon init script and Scala to shared Cluster

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.