Accessing workspace files within cluster init script

glebex
New Contributor II

Greetings all!

I am currently facing an issue while accessing workspace files from the init script.

As it was explained in the documentation, it is possible to place init script inside workspace files (link). This works perfectly fine and init script is being actually executed.

However, it seems that it is not possible to reference a workspace file from the init script itself. E.g. if I placed pyproject.toml file inside my workspace folder (/Workspace/Users/username@email.com/pyproject.toml). Accessing this pyproject.toml within init script fails.

I also tried to debug it a bit and tried to list root directory ("/") and /Workspace directory during init script execution. The result of "ls /" outputs /Workspace folder as visible, however, "ls /Workspace" throws an error:

ls: cannot open directory '/Workspace': Invalid argument

I'm using Azure Databricks with cluster created by me with DB runtime 12.2 LTS ML. Workspace is created as premium and I'm admin on this workspace.

As I see others also are facing the same issue

Regards,

Gleb Smolnik