cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

dbfs access for job clusters and interactive cluster

Praveen2609
New Contributor

Hi All,

I am new to databricks need some understanding for my requirement .

our requirement:

a: we have zip file in azure blob storage and we are bringing that file to dbfs and unzip that file and executing our transformations in multiple steps (3 steps) and every step is in different notebook.so we have used 3 notebook.

b. output of 1st notebook becomes input data for next notebook have use have done through tables created in database in dbfs.

c. Same goes with 3rd notebook also which uses the output produced from 2nd notebook.

d. When we perform these activity through interactive cluster it all goes fine .

question:

1: In production step up right now in org we have job cluster which will spin off for every notebook can we have access to same dbfs from multiple job cluster or we need to mount some storage to handle this scenario.

Regards,

Praveen

2 REPLIES 2

Debayan
Databricks Employee
Databricks Employee

Hi, Databricks enables users to mount cloud object storage to the Databricks File System (DBFS) to simplify data access patterns for users that are unfamiliar with cloud concepts.

The same Azure blob storage can be mounted in multiple clusters.

  • When you create a mount point through a cluster, cluster users can immediately access the mount point. To use the mount point in another running cluster, you must run dbutils.fs.refreshMounts() on that running cluster to make the newly created mount point available for use.

Please refer: https://learn.microsoft.com/en-gb/azure/databricks/dbfs/mounts#--mount-adls-gen2-or-blob-storage-wit...

Anonymous
Not applicable

Hi @praveen rajak​ 

Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group