How do I install a non python/R/maven/jar library into a cluster?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-02-2022 12:28 PM
I'm trying to install a non standard package into the cluster using the init scripts. The package I'm trying to install needs to be downloaded using wget, and uncompressed using tar. Then added to the PATH, or at least I need to know where the downloaded files live.
Here are the contents of my init scripts:
wget https://github.com/COMBINE-lab/salmon/releases/download/v1.8.0/salmon-1.8.0_linux_x86_64.tar.gz
tar xzvf salmon-1.8.0_linux_x86_64.tar.gz
SALMON_PATH=$(readlink -f ./salmon-1.8.0_linux_x86_64/bin/)
export PATH="$SALMON_PATH:$PATH"But this isn't working, at least in a notebook.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-02-2022 04:12 PM
@Bradley (Customer), You can try to provide the directory option while extracting the tar file.
Or you can download the file to the dbfs location and then use the init script to copy it to the path where it should be in the cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-02-2022 04:46 PM
@Bradley (Customer), wget will download the file on the driver, so then it will need to be moved to the filesystem. You can see the distributed file system with %sh ls /dbfs/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-05-2022 09:10 PM
@Bradley Wright Could you please share the error message you are receiving when using the INIT script?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2022 10:13 AM
Thank you for the support. Yes, I was able to find a working solution.
- I placed the files into the distributed file system, dbfs. For others, this can be done manually using the databricks cli, or using the init scripts. In this case I found it easier to use the databricks cli.
- I appended the path to the binary file to the OS environment path. I'm using python so for me it looks like this:
os.environ['PATH'] += ':/dbfs/FileStore/salmon/bin/'And that was it! I stored the file paths to the input data in a dataframe, then used spark to iterate across the rows of the dataframe calling a custom function that calls the binary file.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2022 05:27 PM
I am happy that my suggestion added some pointers for you to resolve the issue.
