topic Re: Using init scripts on UC enabled shared access mode clusters in Data Engineering

Using init scripts on UC enabled shared access mode clusters

ah0896 — Tue, 13 Jun 2023 18:29:25 GMT

I know that UC enabled shared access mode clusters do not allow init script usage and I have tried multiple workarounds to use the required init script in the cluster(pyodbc-install.sh, in my case) including installing the pyodbc package as a workspace library and using that library in the cluster and also using magic commands to install the init script directly in the notebooks but have received errors with both those workarounds. Is there any other ways to use the init scripts in the UC enabled shared access mode cluster?

I have attached the error being received when I try to install the pyodbc library. I am using an admin account as well.

Library installation attempted on the driver node of cluster 0516-171623-3i9on4it and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'pyodbc 4.0.39' --disable-pip-version-check) exited with code 1. WARNING: The directory '/home/libraries/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
ERROR: Invalid requirement: 'pyodbc 4.0.39'

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Wed, 14 Jun 2023 22:06:11 GMT

@Akarsh Hebbar init scripts are only supported on single user access modes, please use single user access mode and try

Re: Using init scripts on UC enabled shared access mode clusters

Anonymous — Thu, 15 Jun 2023 04:37:28 GMT

Hi @Akarsh Hebbar

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Using init scripts on UC enabled shared access mode clusters

ah0896 — Thu, 15 Jun 2023 12:58:55 GMT

@karthik p Thank you for the answer but I do understand that init scripts are not supported on UC shared clusters and are supported only on single user clusters. That's why I wanted to check if there is a workaround to enable the odbc driver on the shared UC cluster.

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Fri, 16 Jun 2023 14:09:06 GMT

@Akarsh Hebbar by any chance r u customizing libraries, if not try to go with workspace library config as shown in below article https://docs.databricks.com/libraries/cluster-libraries.html

Re: Using init scripts on UC enabled shared access mode clusters

ah0896 — Fri, 16 Jun 2023 14:24:17 GMT

@karthik p No I am installing the pyodbc library through the workspace by first installing the library in the workspace and then trying to install it on the cluster and it just fails to install on the cluster and gives the error shown above.

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Fri, 16 Jun 2023 14:35:38 GMT

@Akarsh Hebbar can you go with cluster specific(cluster-installed library) and pick runtime 13.1 or above

Re: Using init scripts on UC enabled shared access mode clusters

ah0896 — Fri, 16 Jun 2023 15:07:39 GMT

@karthik p Yup. that's the only runtime version that supports library installs so the cluster is on version 13.1.

Re: Using init scripts on UC enabled shared access mode clusters

rsenjins — Fri, 30 Jun 2023 11:06:08 GMT

Hello @karthik_p , we encounter this problem. We're having a custom library which we want to use in a convenient way on our multi-user UC cluster. Best case scenario we want to install it by default on the cluster. Although it is not possible to fetch it from our private repository via the libraries interface, and also no init scripts are available. What is the best solution to solve this? We'd rather not install it at every (notebook) instance...

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Fri, 30 Jun 2023 16:21:13 GMT

@aersen never tried this scenario, but can you create global init script and see if it works, as this script is available for all clusters and looks it should support shared mode also (DBR should be 13.1 and above please) https://docs.databricks.com/clusters/init-scripts.html

Re: Using init scripts on UC enabled shared access mode clusters

rsenjins — Fri, 30 Jun 2023 17:46:42 GMT

Thanks for the quick response. Unfortunately the global init scripts did not work for the multi-user UC cluster. It seems quite a challenge to make a customer library available on such clusters. Are you maybe aware whether init scripts for multi user UC clusters are in the pipeline to be developed in further databricks versions >13.1?

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Fri, 30 Jun 2023 19:02:06 GMT

@rsenjins looks response provided by @KKDataEngineer should help you, looks he used workspace library config and use workspace lib in cluster should be able to help.

@KKDataEngineer looks custom library config is not possible for workspace level library, did you make any custom changes to during library config and used in shared access mode please. i can see custom changes may not be supported

Re: Using init scripts on UC enabled shared access mode clusters

KKDataEngineer — Fri, 30 Jun 2023 18:50:00 GMT

@ah0896 it worked when using a workspace level shell scipt file and pointing to its path with workspace scope.

Re: Using init scripts on UC enabled shared access mode clusters

KKDataEngineer — Fri, 30 Jun 2023 19:02:07 GMT

@karthik_p we are still in the preliminary steps on this in our SAAS product. we have not thoroughly tested but have not seen any issues so far. we were able to consistently add libraries to all the clusters with the init scrip in a UC enable workspace.

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Fri, 30 Jun 2023 19:03:54 GMT

Re: Using init scripts on UC enabled shared access mode clusters

KKDataEngineer — Fri, 30 Jun 2023 19:08:01 GMT

No we have not done any custom library configs. You could be right on the limitations.

Re: Using init scripts on UC enabled shared access mode clusters

karthik_p — Sat, 01 Jul 2023 20:23:47 GMT

@Anonymous @Retired_mod can anyone form databricks confirm on above issue please, there seems to be bit conflict on using custom scripts support on shared access mode cluster with unity catalog enabled please

Re: Using init scripts on UC enabled shared access mode clusters

rsenjins — Thu, 06 Jul 2023 08:41:07 GMT

I'm very interested as well! If not possible, then very curious what is the best practice of installing cluster wide custom libraries on shared access mode clusters with unity catalog

Re: Using init scripts on UC enabled shared access mode clusters

praveenVP — Mon, 03 Mar 2025 04:41:01 GMT

Hello all,

Below workaround was efficient to me

1) pyodbc-install.sh is uploaded in a Volume

2) the shared cluster is able to navigate to the Volume to select the init script

3) the Databricks runtime is 15.4 LTS

4) the Allowlist has been updated to allow the init script

5) the cluster spins up with no problem

6) the pyodbc==5.2.0 library has been installed (although it made no difference) but

If you are using from ADF use below volumes option and provide ADF SP access to volume path