cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

horatiug
by New Contributor III
  • 858 Views
  • 0 replies
  • 0 kudos

Can the databricks_mount timeout be changed. ?

I am using terrafom to do databricks workspace configuration and while mounting 6 buckets if duration of mount is bigger than 20 min I get timeout. Is it possible to change the timeout ? thanksHoratiu

  • 858 Views
  • 0 replies
  • 0 kudos
augustin
by New Contributor II
  • 4621 Views
  • 5 replies
  • 5 kudos

Mount an uncrypted AWS EFS in AWS Databricks

Hi,I want to mount an uncrypted AWS EFS in AWS Databricks. When I do:mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-abcdef.efs.region.amazonaws.com:/ /mnt/efs-uncryptedI get this error:mount.nfs4: moun...

  • 4621 Views
  • 5 replies
  • 5 kudos
Latest Reply
Andrei_Radulesc
Contributor III
  • 5 kudos

"To support NFS under LXC, some of the apparmor protections need to be lifted." (see https://theorangeone.net/posts/mount-nfs-inside-lxc/)

  • 5 kudos
4 More Replies
LukaszJ
by Contributor III
  • 3772 Views
  • 5 replies
  • 4 kudos

Resolved! Mount Azure Blob Storage with Cluster access control

Hello.I want to mount and share for the one group the container from Azure Blob Storage (It could be simple blob storage or Azure Data Lake Storage gen 2). But I am not able to do it because I am using Cluster with Table Access Control.This is my cod...

  • 3772 Views
  • 5 replies
  • 4 kudos
Latest Reply
LukaszJ
Contributor III
  • 4 kudos

I have a good solution to the problem:I am using Python library.There are some documentation.Topic to be closed.Best regards,Łukasz

  • 4 kudos
4 More Replies
kkumar
by New Contributor III
  • 1461 Views
  • 2 replies
  • 2 kudos

ADLS Gen 2 Delta Tables memory allocation

if i mount a gen2(ADLS 1) to another gen2(ADLS2) account and create a delta table on ADLS2 will it copy the data or just create something link External table.i don't want to duplicate the the data.

  • 1461 Views
  • 2 replies
  • 2 kudos
Latest Reply
Pat
Honored Contributor III
  • 2 kudos

Hi @keerthi kumar​ ,so basically you can CREATE EXTERNAL TABLES on top of the data stored somewhere - in your case ADLS. Data won't be copied, it will stay where it is, by creating external tables you are actually storing the metadata in your metasto...

  • 2 kudos
1 More Replies
StuartParker188
by New Contributor III
  • 6292 Views
  • 5 replies
  • 5 kudos

How to handle java.io.Exception in python notebook

I'm attempting to mount a volume using dbutils.fs.mount in a python workbookin the exception handling for this statement, I have found an exception that doesn't get caught using the standard try/except handlingfor example, if passing through a contai...

  • 6292 Views
  • 5 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Stuart Parker​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 5 kudos
4 More Replies
niels
by New Contributor III
  • 1604 Views
  • 2 replies
  • 0 kudos

Azure SA mounted but can't load files

I am attempting to load an excel file that's located in a blob storage that I've mounted. In the first cell, when I use the dbutils.fs.ls command, I can see the file I want to load. However, when I try to actually load it, it can't find the file. It ...

  • 1604 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Niels Ota​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
rahul3
by New Contributor
  • 2747 Views
  • 1 replies
  • 1 kudos

Facing mount/unmount issue while running same job parallelly with scala.

 Using above configuration in cluster, when I run databricks job parallelly with multiple request at a same time, then I am getting mount/unmount issue. For an example : When I make three request to databricks job , it run 3 jobs parallelly but somet...

image.png image.png
  • 2747 Views
  • 1 replies
  • 1 kudos
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

hi @rahul upadhyay​ are you using the same mount path /mnt/rahul in all the 3 jobs? Could you please add the full error message?

  • 1 kudos
Ashley1
by Contributor
  • 2512 Views
  • 5 replies
  • 1 kudos

Resolved! Can ADLS be mounted in DBFS using only ADLS account key?

I realise this is not an optimal configuration but I'm trying to pull together a POC and I'm not at the point that I wish to ask the AAD admins to create an application for OAuth authentication.I have been able to use direct references to the ADLS co...

  • 2512 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hey there @Ashley Betts​ Thank you for posting your question. And you found the solution.This is awesome!Would you be happy to mark the answer as best so that other members can find the solution more quickly?Cheers!

  • 1 kudos
4 More Replies
matt_t
by New Contributor
  • 3334 Views
  • 2 replies
  • 1 kudos

Resolved! S3 sync from bucket to a mounted bucket causing a "[Errno 95] Operation not supported" error for some but not all files

Trying to sync one folder from an external s3 bucket to a folder on a mounted S3 bucket and running some simple code on databricks to accomplish this. Data is a bunch of CSVs and PSVs.The only problem is some of the files are giving this error that t...

  • 3334 Views
  • 2 replies
  • 1 kudos
Latest Reply
Atanu
Databricks Employee
  • 1 kudos

@Matthew Tribby​  does above suggestion work. Please let us know if you need further help on this. Thanks.

  • 1 kudos
1 More Replies
Bhanu1
by New Contributor III
  • 4248 Views
  • 3 replies
  • 6 kudos

Resolved! Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?

  • 4248 Views
  • 3 replies
  • 6 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 6 kudos

Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time. you can have for example following configuration:In Azure you can have 2 databricks workspace,cluster in every workspace can have env variable is...

  • 6 kudos
2 More Replies
Confused
by New Contributor III
  • 4095 Views
  • 6 replies
  • 1 kudos

Hi Guys Is there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace...

Hi GuysIs there any documentation on where the /databricks-datasets/ mount is actually served from?We are looking at locking down where our workspace can reach out to via the internet and as it currently stands we are unable to reach this.I did look ...

  • 4095 Views
  • 6 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hello Mat, Thanks for letting us know. Would you be happy to mark your answer as best if that will solve the problem for others? That way, members will be able to find the solution more easily.

  • 1 kudos
5 More Replies
Prabakar
by Databricks Employee
  • 1960 Views
  • 0 replies
  • 2 kudos

Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regions that are disabled by default. You must first enabl...

Accessing the regions that are disabled by default in AWS from Databricks.In AWS we have 4 regions that are disabled by default. You must first enable it before you can create and manage resources. The following Regions are disabled by default:Africa...

  • 1960 Views
  • 0 replies
  • 2 kudos
Erik
by Valued Contributor III
  • 4937 Views
  • 6 replies
  • 2 kudos

Run more than nr-of-cores concurrent tasks.

We are using the terraform databricks provier, which is starting a cluster and checking every mount (since there is no mount rest API!). Each mount takes 20 seconds to check, and 99.9% of that time is idle waiting, and it starts a job per mount. If w...

  • 4937 Views
  • 6 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

hi @Erik Parmann​ ,It is possible to do, but you might need to also enable dynamic allocation at the cluster level to be able to make sure your settings are apply at cluster creation . You can find more details here. As best practice, we do not recom...

  • 2 kudos
5 More Replies
PraveenKumar188
by New Contributor
  • 3057 Views
  • 2 replies
  • 2 kudos

Resolved! Is is possible to Mount multiple ADLS Gen2 Storage paths in single workspace

Hello Experts,We are looking on feasibility of mounting more that one ADLS Gen2 storages on a single workspace of databricks.Best RegardsPraveen

  • 3057 Views
  • 2 replies
  • 2 kudos
Latest Reply
Erik
Valued Contributor III
  • 2 kudos

Yes, its possible, we are doing it. Just mount them to different folders like @Werner Stinckens​ is saying.

  • 2 kudos
1 More Replies
stramzik
by New Contributor II
  • 1450 Views
  • 1 replies
  • 1 kudos

Unable to mount datalake gen1 to databricks

I was mounting the Datalake Gen1 to Databricks for accessing and processing files, The below code was working great for the past 1 year and all of a sudden I'm getting an errorconfigs = {"df.adl.oauth2.access.token.provider.type": "ClientCredential"...

  • 1450 Views
  • 1 replies
  • 1 kudos
Latest Reply
stramzik
New Contributor II
  • 1 kudos

bumping up the thread

  • 1 kudos
Labels