topic Re: Accessing workspace files within cluster init script in Data Engineering

Accessing workspace files within cluster init script

glebex — Tue, 13 Jun 2023 12:56:16 GMT

Greetings all!

I am currently facing an issue while accessing workspace files from the init script.

As it was explained in the documentation, it is possible to place init script inside workspace files (link). This works perfectly fine and init script is being actually executed.

However, it seems that it is not possible to reference a workspace file from the init script itself. E.g. if I placed pyproject.toml file inside my workspace folder (/Workspace/Users/username@email.com/pyproject.toml). Accessing this pyproject.toml within init script fails.

I also tried to debug it a bit and tried to list root directory ("/") and /Workspace directory during init script execution. The result of "ls /" outputs /Workspace folder as visible, however, "ls /Workspace" throws an error:

ls: cannot open directory '/Workspace': Invalid argument

I'm using Azure Databricks with cluster created by me with DB runtime 12.2 LTS ML. Workspace is created as premium and I'm admin on this workspace.

As I see others also are facing the same issue

Regards,

Gleb Smolnik

Re: Accessing workspace files within cluster init script

-werners- — Tue, 13 Jun 2023 13:08:58 GMT

here is a similar topic.

We haven't figured it out yet but it might be helpful for you.

Re: Accessing workspace files within cluster init script

Anonymous — Wed, 14 Jun 2023 07:45:52 GMT

@Gleb Smolnik :

The init script runs on the cluster nodes before the notebook execution, and it does not have direct access to workspace files.

The documentation you mentioned refers to placing the init script inside a workspace file, which means you can store the script itself in a file within the Databricks workspace. However, it doesn't grant direct access to other workspace files from within the init script.

To access a workspace file within the init script, you can consider using the Databricks CLI or Databricks API to retrieve the file and then copy or read it on the cluster nodes during the init script execution.

Re: Accessing workspace files within cluster init script

glebex — Wed, 14 Jun 2023 08:31:37 GMT

Hey @Suteja Kanuri,

Thanks for your answer. I understand your point. However, I could imagine the scenario, when init script acts as an orchestrator, executing other shell scripts in a desired order. The documentation article I referenced (at least how I interpreted it) allows placing init script into workspace files, kind of implying that other files will be accessible during init script execution too (which is not the case).

Anyways, I will try to figure it out with the suggestions you provided. It will be obviously nice to have workspace files mounted to databricks cluster before init script execution (not sure is it a part of a feature roadmap, so just a suggestion).

Re: Accessing workspace files within cluster init script

jacob_hill_prof — Tue, 20 Jun 2023 20:44:01 GMT

@Gleb Smolnik You might also want to try cloning a github repo in your init script and then storing dependencies like requirements.txt files and other init scripts there. By doing this you can pull a whole slew of init scripts to be utilized in your cluster dynamically from a versioned source.

init.sh

git clone <github repo url>my_repo.git
git -C ./my_repo checkout common_cluster_init # checkout non-main branches
pip install -r ./my_repo/init/dbricks_clusters/requirements.txt # use scripts in the repo

Re: Accessing workspace files within cluster init script

FRG96 — Fri, 07 Jul 2023 05:56:26 GMT

Link isn't working anymore

Re: Accessing workspace files within cluster init script

FRG96 — Fri, 07 Jul 2023 07:28:58 GMT

Hi @Anonymous @glebex

I want to use the Databricks Workspace export REST API using curl in the init script to download a workspace file locally.
What's the recommended way to pass the Databricks Instance URL and the API Token to the init script execution context?

Re: Accessing workspace files within cluster init script

Nitya — Mon, 10 Jul 2023 06:07:08 GMT

When we are using databricks CLI - it didn't copied .txt file and in the another workaround using databricks API, it is using dbfs there is no API regarding Workspace FIle. Just wanted to check if there is an another way to accessing workspace files within cluster init script.