topic Re: How to access a jar file stored in Databricks Workspace ? in Data Engineering

How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 10:52:10 GMT

Hi All,

We have a couple of jars stored in a workspace folder.

We are using init scripts to copy the jars in the workspace to the /databricks/jars path.

The init scripts do not seem to be able to find the files.

The scripts are failing saying the files could not be found.

```bash

#!/bin/bash

cp /Workspace/jars/file_name.jar /databricks/jars/

cp /Workspace/jars/file_name.jar /databricks/databricks-hive/

```

Can you please let me know if this is even possible ?

What is the correct path for a file in a Workspace Folder.

Thank you...

Edit:

Have tried file path with and without /Workspace - both are failing saying file is not available.
I have also tried sleep for up to 2 minutes and the files are still not available.

Would be nice if someone from databricks can confirm if binary files from Workspace such as jars are accessible via init scripts, if yes what would their path be like ?

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 11:08:25 GMT

that seems to be ok. Probably the file system is not yet mounted when you do the copy operations.

What is your use case of copying jars in an init script? There might be alternatives.

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 11:16:15 GMT

Thank you for your response...

We just have an inbuilt jar file that we have in DBFS and move to above mentioned paths as part of the init scripts...

As part of the process to move the init scripts to a workspace location, we are just trying to see if we can have the jars organized in Workspace and use similar init script to move it...We also planned on having a few files copied similarly...

Follow up Questions:

Should the workspace file path begin with `/Workspace/jars/file_name.jar` or just `/jars/file_name.jar`
What would be the best option to work around the file system delay ? is it possible to check if it is up or a sleep command ?

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 11:19:54 GMT

/Workspace is a directory (you can check by using the %sh magic command in a notebook)
Well, the way I use my jars is by installing them on a cluster as a library. By then everything is mounted and the jars can be found. My jars reside in the /Filestore/jars directory and do not move or are not being copied.

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 11:25:30 GMT

Thank you for your response...

The reason for the question is when I mention an init script as part of the cluster config, there the word /Workspace is not used. On the other hand %sh path uses that word.
Currently we also have it placed similarly, but when I need to show the files/jars to someone it is a pain. Showing from the workspace is even easier. Also some arbitrary files can be pushed to git easily from Workspace.

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 11:28:10 GMT

I see, but isn't it easier to share the source code in git instead of a jar?

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 11:31:13 GMT

Code is in git only, we have blackbox jars provided by different teams and used as part of the code. These are the ones we currently have in DBFS and plan to move to workspace.

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 11:35:06 GMT

I still struggle to understand why you copy them, sorry.

A jar is an artifact. You import them and use them in your program. For that you do not need a copy.

If you need to know what is in the jar, go to git and look at the class code.

Probably I am missing something.

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 11:43:19 GMT

No issues...I will try to explain...

We have several teams...One of them produces a jar file whose logic is black box to us. We only use it. We are not aware of its contents. It is a carry forward from old legacy code.

To use the jar file, we copy it to the drive so that our code can reference the classes inside the jar file. To do that we copy the jar file to the path where databricks copies all the jars - you can see it some environmental variable - I do not remember...

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 11:55:16 GMT

ok i get it.

is the databricks-cli an option? Because my guess is that Workspace is not available during init script( you do use a cluster-scoped init script right?).

Or you can put the jars to be copied in a /databricks subfolder like /zzjars as /databricks should exist.

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 12:10:44 GMT

Are you saying the Workspace path will not be available even if I sleep for a minute or so ? Can you please confirm ? This is will save me a lot of effort if it is not going to load. Ideally it should not be the case right since Databricks is proposing that the init scripts should be maintained as part of Workspace meaning Workspace should logically be available first before the init scripts execute right ?

Any idea on the order of availability ?

Driver Node

Mount DBFS

Mount Workspace

Init Scripts

Other Mounts ?

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 12:35:45 GMT

Hm, you are right. If you need to put init scripts in the workspace, it should be available already.

So it seems that you need to get the path correct.

Can you try with the /Worspace dir in the path (as you mentioned the script does not use it)?

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 13:14:06 GMT

Have tried it both ways with and without /Workspace...the files do not seem to be available...Also have tried sleeping for up to 2 minutes to see if it is delay in mounting issue...I just think binary files such as jars are not supported, would be nice if someone from databricks can confirm...

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 13:17:50 GMT

Can you write the directory structure to a file in the init script? like that you can take a look if you are missing a part of the path.

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 07 Jun 2023 13:35:19 GMT

I did...A very simple init script

Tried with and without Workspace....

```bash

#!/bin/bash

list_of_files=$(ls /Workspace/jars/)

printf "$list_of_files" > /dbfs/FileStore/temp_files/init_script_error.txt

```

In both the cases got the similar error message...

```bash

ls: cannot access '/Workspace/jars/': Invalid argument

```

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 07 Jun 2023 13:39:55 GMT

perhaps using tree -d? (probably you have to install it first)

or there is also something as the web terminal but I have never used that

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Tue, 13 Jun 2023 06:24:28 GMT

Thank you for your response @Kaniz Fatma

The issue is still not resolved.

I was hoping someone from Databricks would be able to help. Please help if possible.

Steps to Reproduce:

Have two folders in Workspace - init_scripts and jars.
Have any jar file in the jars folder.
Have an init script in the init_scripts folder that copies the jar file from the jars folder into the /databricks/databricks-hive/ folder in the cluster.

```bash

#!/bin/bash

cp /Workspace/jars/file_name.jar /databricks/jars/

cp /Workspace/jars/file_name.jar /databricks/databricks-hive/

```

Configure the init script in the workspace in the Cluster configuration.

Expected:

Cluster needs to start
The jar has to be copied to the /databricks/databricks-hive/ folder.

Current Behaviour:

The cluster is not starting.
Init script is failing stating that the source file is not available.

Re: How to access a jar file stored in Databricks Workspace ?

Anonymous — Wed, 14 Jun 2023 06:56:12 GMT

Hi @Bharath Kumar Ramachandran

We haven't heard from you since the last response from @Werner Stinckens . Kindly share the information with us, and in return, we will provide you with the necessary solution.

Thanks and Regards

Re: How to access a jar file stored in Databricks Workspace ?

ranged_coop — Wed, 14 Jun 2023 07:03:05 GMT

Hi @Vidula Khanna , thank you for your response. I had also responded to a similar message from @Kaniz Fatma . Anyways the issue is still not resolved.

The issue is still not resolved.

I was hoping someone from Databricks would be able to help. Please help if possible.

Steps to Reproduce:

Have two folders in Workspace - init_scripts and jars.
Have any jar file in the jars folder.
Have an init script in the init_scripts folder that copies the jar file from the jars folder into the /databricks/databricks-hive/ folder in the cluster.

```bash

#!/bin/bash

cp /Workspace/jars/file_name.jar /databricks/jars/

cp /Workspace/jars/file_name.jar /databricks/databricks-hive/

```

Configure the init script in the workspace in the Cluster configuration.

Expected:

Cluster needs to start without any error.
The jar has to be copied to the /databricks/databricks-hive/ folder.

Current Behaviour:

The cluster is not starting.
Init script is failing stating that the source file is not available.

Things already tried:

Have tried file path with and without /Workspace - both are failing saying file is not available.
I have also tried sleep for up to 2 minutes and the files are still not available.
Have tried an init script that would just list the files in the path and print to a file. Even listing fails stating file/path is not available.

Re: How to access a jar file stored in Databricks Workspace ?

-werners- — Wed, 14 Jun 2023 07:54:55 GMT

Someone has the same issue as you and well, it is not possible:

https://community.databricks.com/s/question/0D78Y000007uNLYSA2/detail