Mount Workspace to Docker container
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-23-2023 05:58 PM
Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?
If I create a cluster without a Docker image, the `!ls` command in Databricks notebook returns `/Workspace` folder that contains user folders, as well as their mounted Git repositories.
Is there a way to make this folder available when running a cluster with a Docker image?
- Labels:
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-02-2023 04:12 PM
Hello:
Thanks for contacting Databricks Support!
I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem.
When you use custom containers in Databricks, your notebooks still run inside the Databricks workspace, and you can still access DBFS and other Databricks-specific features. The container is essentially providing the runtime environment, but you're still operating within the Databricks ecosystem. So, when you execute commands within a notebook, it's executed within that Databricks context, and you should still have access to the /Workspace and other Databricks-specific directories.
Regards,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2023 09:19 AM - edited 11-22-2023 09:21 AM
So, when you execute commands within a notebook, it's executed within that Databricks context, and you should still have access to the /Workspace and other Databricks-specific directories.
This does not appear to be true. When I try to open any path in /Workspace from a python notebook, on a cluster running, for example, the image projectglow/databricks-glow:10.4, I get "No such file or directory" errors.

