My Global Init Script doesn't run with the new(er) LTS version 13.3. It runs great on 12.2LTS

[Benedetta]

Hey Databricks,

     Seems like you changed the way Global Init Scripts work.  How come you changed it? My Global Init Script runs great on 12.2LTS but not on the new(er) LTS version 13.3. We don't have Unity Catalog turned on. What's up with that? Are Global Init scripts going away for good?

Thank you

Benedetta

[Kaniz_Fatma]

Hi @Benedetta, Hello! It appears you’ve noticed some changes in the way Global Init Scripts function in Databricks. Let me provide some insights into this:

1. Global Init Scripts:

   • These scripts run on every cluster created within your workspace.
   • They are useful when you want to enforce organization-wide library configurations or implement security measures.
   • Only workspace admins can create global init scripts, and they can be set up using either the UI or REST API.
   • Keep in mind that because global init scripts run on all clusters, there are potential impacts to consider:
     • It’s easy to add libraries or make other modifications that cause unanticipated effects.
     • Whenever possible, use cluster-scoped init scripts instead.
     • Any user who creates a cluster and enables cluster log delivery can view the stderr and stdout output from global init scripts. So, ensure that your global init scripts don’t output sensitive information.
     • You can troubleshoot global init scripts by configuring cluster log delivery and examining the init ...

2. Cluster-Scoped Init Scripts:

   • Databricks recommends configuring all init scripts as cluster-scoped and managing them across your workspace using cluster policies.
   • Cluster-scoped init scripts allow you to define configurations specific to individual clusters, avoiding unintended global impacts.
   • If you’re using Databricks Runtime 13.3 LTS or above, you can also explore Unity Catalog clusters wi...².
   • To migrate cluster-scoped init scripts from DBFS to Unity Catalog volumes, you can use the provided ...³.

3. Unity Catalog:

   • Since you mentioned that you don’t have Unity Catalog turned on, it’s essential to understand that global init scripts are not run on model serving clusters.
   • As for the future of global init scripts, there’s no indication that they’re going away for good. However, Databricks continues to evolve its platform, so staying informed about updates is crucial.

Remember to restart your clusters after making any changes to init scripts for them to take effect. If you have further questions or need assistance, feel free to ask! 😊

[Benedetta]

Hi Kaniz,

     Thanks for your reply. I still don't understand if my pyodbc global init script is supposed to run on LTS 13.3. If so.....it doesn't. The connection to the server fails. In the past when upgrading to the latest LTS runtime, I don't recall the init scripts failing.  Could you please confirm if my init script should run on the newer 13.3 LTS version, as it does on the LTS 12.2 version?

Thank you!

Benedetta

[ChloeBors · Accepted solution]

Hello Benedetta,

I had the same issue with an odbc init script.

Within the global script it was using an ubuntu version (16.04).
After I upgraded this version in the script to 22.04 it worked! (that 22.04 version also shows on Databricks Runtime 13.3 LTS | Databricks on AWS under System Environment)

I hope this will help you!

[Benedetta]

Thank you ChloeBors - I tried upgrading the ubuntu version per your suggestion but got a new error "Can t open lib ODBC Driver 17 for SQL Server file not found 0 SQLDriverConnect". I tried modifying this line to 18:  ACCEPT_EULA=Y apt-get install msodbcsql17 but I got the same error about 17. Not sure what to do next.