topic Re: How do I install a non python/R/maven/jar library into a cluster? in Machine Learning

How do I install a non python/R/maven/jar library into a cluster?

Bradley — Thu, 02 Jun 2022 19:28:54 GMT

I'm trying to install a non standard package into the cluster using the init scripts. The package I'm trying to install needs to be downloaded using wget, and uncompressed using tar. Then added to the PATH, or at least I need to know where the downloaded files live.

Here are the contents of my init scripts:

wget https://github.com/COMBINE-lab/salmon/releases/download/v1.8.0/salmon-1.8.0_linux_x86_64.tar.gz
tar xzvf salmon-1.8.0_linux_x86_64.tar.gz
SALMON_PATH=$(readlink -f ./salmon-1.8.0_linux_x86_64/bin/)
export PATH="$SALMON_PATH:$PATH"

But this isn't working, at least in a notebook.

Re: How do I install a non python/R/maven/jar library into a cluster?

Prabakar — Thu, 02 Jun 2022 23:12:10 GMT

@Bradley (Customer), You can try to provide the directory option while extracting the tar file.

Or you can download the file to the dbfs location and then use the init script to copy it to the path where it should be in the cluster.

Re: How do I install a non python/R/maven/jar library into a cluster?

Anonymous — Thu, 02 Jun 2022 23:46:57 GMT

@Bradley (Customer), wget will download the file on the driver, so then it will need to be moved to the filesystem. You can see the distributed file system with %sh ls /dbfs/

Re: How do I install a non python/R/maven/jar library into a cluster?

User16764241763 — Mon, 06 Jun 2022 04:10:18 GMT

@Bradley Wright Could you please share the error message you are receiving when using the INIT script?

Re: How do I install a non python/R/maven/jar library into a cluster?

Bradley — Mon, 20 Jun 2022 17:13:14 GMT

Thank you for the support. Yes, I was able to find a working solution.

I placed the files into the distributed file system, dbfs. For others, this can be done manually using the databricks cli, or using the init scripts. In this case I found it easier to use the databricks cli.
I appended the path to the binary file to the OS environment path. I'm using python so for me it looks like this:

os.environ['PATH'] += ':/dbfs/FileStore/salmon/bin/'

And that was it! I stored the file paths to the input data in a dataframe, then used spark to iterate across the rows of the dataframe calling a custom function that calls the binary file.

Re: How do I install a non python/R/maven/jar library into a cluster?

Prabakar — Sat, 16 Jul 2022 00:27:46 GMT

I am happy that my suggestion added some pointers for you to resolve the issue.