topic Re: Mount Workspace to Docker container in Data Engineering

Mount Workspace to Docker container

ivanychev — Thu, 24 Aug 2023 00:58:34 GMT

Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?

If I create a cluster without a Docker image, the `!ls` command in Databricks notebook returns `/Workspace` folder that contains user folders, as well as their mounted Git repositories.

Is there a way to make this folder available when running a cluster with a Docker image?

Re: Mount Workspace to Docker container

User16539034020 — Thu, 02 Nov 2023 23:12:42 GMT

Hello:

Thanks for contacting Databricks Support!

I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem.

When you use custom containers in Databricks, your notebooks still run inside the Databricks workspace, and you can still access DBFS and other Databricks-specific features. The container is essentially providing the runtime environment, but you're still operating within the Databricks ecosystem. So, when you execute commands within a notebook, it's executed within that Databricks context, and you should still have access to the /Workspace and other Databricks-specific directories.

Regards,

Re: Mount Workspace to Docker container

IgorP-NVTA — Wed, 22 Nov 2023 17:21:01 GMT

So, when you execute commands within a notebook, it's executed within that Databricks context, and you should still have access to the /Workspace and other Databricks-specific directories.

This does not appear to be true. When I try to open any path in /Workspace from a python notebook, on a cluster running, for example, the image projectglow/databricks-glow:10.4, I get "No such file or directory" errors.