How install a library that is under the /Workspace/Shared/ directory using the init.sh script in a cluster?

[juned]

I would like to install a library that is under the /Workspace/Shared/ directory using the init.sh script in a cluster.

How to access the /Workspace/Shared/ folder in shell? This page only shows how to access manually but doesn't show how to access in a shell script - Workspace libraries - Azure Databricks | Microsoft Learn

The documentation - Cluster node initialization scripts - Azure Databricks | Microsoft Learn provides only databricks-cli library install command for libraries that are in the dbfs:/FileStore/JAR directory but unable to navigate to /Workspace/Shared/ folder.

[Anonymous]

@Juned Mala​ :

To access the /Workspace/Shared/ directory in a shell script, you can use the Databricks CLI command databricks fs to navigate the Databricks File System (DBFS).

Here's an example of how to install a library from /Workspace/Shared/ using the init.sh script.

#!/bin/bash
 
# Install library from /Workspace/Shared/
databricks fs cp dbfs:/Workspace/Shared/my_library-1.0.jar /tmp/my_library.jar
databricks libraries install --cluster-id <cluster-id> --jar /tmp/my_library.jar
 
# Add other cluster initialization steps here

In this example, we use databricks fs cp to copy the library from /Workspace/Shared/ to /tmp/ in the DBFS. Then we use databricks libraries install to install the library on the cluster, specifying the path to the library in /tmp/.

Note that you'll need to replace <cluster-id> with the ID of your Databricks cluster. You can find the cluster ID by navigating to the cluster in the Databricks workspace and looking at the URL, which will contain a string like clusterId=1234-567890-abcdefg.

[Anonymous]

Hi @Juned Mala​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!