topic Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..." in Data Engineering

Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

jborn — Thu, 11 Jan 2024 17:14:55 GMT

I recently had an Azure Databricks setup done behind a VPN. I'm trying to connect to my Azure Storage Account Gen 2 Using the following code I haven't been able to connect and keep getting stuck on reading the file. What should I be checking?

#initialize the source path and create the bronze df SourceFilePath = f"abfss://" + PersistedContainer + "@" + AzureBlobStorageAccountName + ".dfs.core.windows.net/" + PersistedDirectory + "/" + PersistedFilename print(SourceFilePath) #Path prints out as expected spark.conf.set(f"fs.azure.account.key." + AzureBlobStorageAccountName + ".dfs.core.windows.net","Key removed") dfBronze = spark.read.load(SourceFilePath,format='csv',header = True).cache()

When I get to the spark.read.load line I get stuck on "Running Command..."

Any commands I can use to get debug information on what is happening?

Using the shell command below resolves to the correct IP.

%sh nslookup AzureBlobStorageAccountName.dfs.core.windows.net

Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

Debayan — Fri, 12 Jan 2024 15:57:29 GMT

Hi, Could you please tell us what is the error code you are receiving?

Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

jborn — Fri, 12 Jan 2024 16:12:13 GMT

Would love to, but no matter how long I keep the notebook running it hangs on the spark.read.load line with "Running Command..." No error message ever comes up.

Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

Debayan — Thu, 18 Jan 2024 19:09:51 GMT

Hi,

It keeps on running, does it happen with other notebooks? What about the cluster? Does it show any error in the driver logs? Can you try with other DBR versions or node types?

Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

jborn — Thu, 18 Jan 2024 19:42:34 GMT

None of them answered my question

Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

jborn — Thu, 18 Jan 2024 19:44:24 GMT

I was unable to find any error message in the driver logs. Not sure what you meant by the DBR versions or node types?

Re: Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

jborn — Thu, 18 Jan 2024 19:45:45 GMT

I ended up opening a ticket with Microsoft support about this issue and they walked us through the debugging on the issue. In the end the route table was not attached to the subnet. Once attached everything worked.