Databricks Community

a2_ish · ‎10-04-2022

I have tried below code to write data in a delta table and save the delta files in a sink. I tried using azure storage as sink but I get error as not enough access, I can confirm that I have enough access to azure storage, however I can run the below code with path as users/ankit.kumar@databricks.com/datadash and it runs , however I am not sure where to find this path physically where can I see the delta files which has been created.

%sql
USE CATALOG hive_metastore;
CREATE DATABASE IF NOT EXISTS demo_db;
USE DATABASE demo_db;
 
#This gives error as shaded.databricks.org.apache.hadoop.fs.azure.AzureException: #hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This request is not #authorized to perform this operation using this permission.
path = "wasbs://abcstoragecontainer@azlogs.blob.core.windows.net"
(bronzeDF.writeStream
  .format('delta') 
  .outputMode("append") 
  .trigger(once=True) 
  .option("mergeSchema", "true")
  .option('checkpointLocation', path+"/bronze_checkpoint")
  .toTable('turbine_bronze')) #.start(path + "/turbine_bronze"))
 
 
#This path works but where is this path located..where can I find it
 
path = "/Users/ankit.kumar@databricks.com/demo_db"
(bronzeDF.writeStream
  .format('delta') 
  .outputMode("append") 
  .trigger(once=True) 
  .option("mergeSchema", "true")
  .option('checkpointLocation', path+"/bronze_checkpoint")
  .toTable('turbine_bronze')) #.start(path + "/turbine_bronze"))

Hubert-Dudek · ‎10-12-2022

It is easier to mount ADLS as a folder as explained here:

https://community.databricks.com/s/feed/0D53f00001eQGOHCA4

Anonymous · ‎11-02-2022

Hi @Ankit Kumar

Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!