Databricks Community

Matt101122 · 11-29-2022

We are attempting to setup Unity Catalog and our security team is requesting justification on why this level of access is required. Why must the first Azure Databricks account admin must be an Azure Active Directory Global Administrator at the time t...

Matt101122 · 10-12-2022

We are having an issue with calls to /api/2.1/jobs/create taking between 30 and 60 seconds to complete. Calls to /api/2.1/jobs/list complete as expected in about 6 seconds so the issue seems to be only with the job create API. This seems to only be h...

Matt101122 · 10-11-2022

I'm extracting data from a custom format by day of month using a 32 core executor. I'm using rdds to distribute work across cores of the executor. I'm seeing an intermittent issue where for a run sometimes I see 31 cores being used as expected and ot...

Matt101122 · 01-10-2023

@Hubert Dudek Do you know if addFile should work with abfss:// path? Trying to add a file from azure data lake with external location in unity catalog.

Matt101122 · 01-10-2023

@Nandini Raja I did something similar by using shutil instead of the dbutils. This worked for copying many local files to Azure Storage in paralell. However, the issue I'm having now is finding a Unity Catalog friendly solution as mounting Azure Sto...

Matt101122 · 10-20-2022

@Debayan Mukherjee the issue has been resolved! Basically all calls to a domain name were slow.We determined that traffic couldn't get to our primary azure dns so it was using the corporate dns on-prem. Now that a networking issue has been resolved ...

Matt101122 · 10-18-2022

@Debayan Mukherjee I understand that the create endpoint would take longer but there is a significant difference between our standard workspace and our premium workspace. The standard is much faster whereas premium is slower... is there anything els...

Matt101122 · 10-13-2022

I may have figured this out! I'm explicitly setting the number of slices instead of using the default.days_rdd = sc.parallelize(days_to_process,len(days_to_process))

Databricks Community

User Stats

User Activity

Unity Catalog Setup: Why must the first Azure Databricks account admin must be an Azure Active Directory Global Administrator at the time that they first log in to the Azure Databricks account console?

Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

why aren't rdds using all available cores of executor?

Re: How to process files from the internet in databricks? "spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get&quo...

Re: Pyspark: You cannot use dbutils within a spark job

Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Re: Why are calls to /api/2.1/jobs/create taking 30-60 seconds to complete?

Re: why aren't rdds using all available cores of executor?