Connecting an Azure Databricks to Azure Gen 2 storage stuck on "Running Command..."

[jborn]

I recently had an Azure Databricks setup done behind a VPN.  I'm trying to connect to my Azure Storage Account Gen 2  Using the following code I haven't been able to connect and keep getting stuck on reading the file.  What should I be checking?  

#initialize the source path and create the bronze df 
SourceFilePath = f"abfss://" + PersistedContainer + "@" + AzureBlobStorageAccountName + ".dfs.core.windows.net/" + PersistedDirectory + "/" + PersistedFilename
print(SourceFilePath)
#Path prints out as expected
spark.conf.set(f"fs.azure.account.key." + AzureBlobStorageAccountName + ".dfs.core.windows.net","Key removed")
dfBronze = spark.read.load(SourceFilePath,format='csv',header = True).cache()

When I get to the spark.read.load line I get stuck on "Running Command..." 

Any commands I can use to get debug information on what is happening?

Using the shell command below resolves to the correct IP.

%sh nslookup AzureBlobStorageAccountName.dfs.core.windows.net​

[Debayan]

Hi, Could you please tell us what is the error code you are receiving? 

[jborn]

Would love to, but no matter how long I keep the notebook running it hangs on the spark.read.load line with "Running Command..." No error message ever comes up.

[Debayan]

Hi,

It keeps on running, does it happen with other notebooks? What about the cluster? Does it show any error in the driver logs? Can you try with other DBR versions or node types? 

[jborn]

I was unable to find any error message in the driver logs.  Not sure what you meant by the DBR versions or node types?

[jborn]

None of them answered my question

[jborn · Accepted solution]

I ended up opening a ticket with Microsoft support about this issue and they walked us through the debugging on the issue.  In the end the route table was not attached to the subnet.  Once attached everything worked.