Where are delta lake files stored by given path?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-04-2022 03:31 AM
I have below code which works for the path below but fails for path = azure storage account path. i have enough access to write and update the storage account. I would like to know what wrong am I doing and the path below which works , how can i physically access it and see the delta files stored?
How is it done?
%sql
USE CATALOG hive_metastore;
CREATE DATABASE IF NOT EXISTS demo_db;
USE DATABASE demo_db;
#write the stream to a sink
#this fails with error shaded.databricks.org.apache.hadoop.fs.azure.AzureException: #hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This request is not #authorized to perform this operation using this permission.
path = "wasbs://astoragecontainer@azauditlogs.blob.core.windows.net"
(bronzeDF.writeStream
.format('delta')
.outputMode("append")
.trigger(once=True)
.option("mergeSchema", "true")
.option('checkpointLocation', path+"/bronze_checkpoint")
.toTable('turbine_bronze')) #.start(path + "/turbine_bronze"))
#this path works but where can I find this path /Users/ankit.kumar@xyz.com/demo_db
path = "/Users/ankit.kumar@xyz.com/demo_db"
(bronzeDF.writeStream
.format('delta')
.outputMode("append")
.trigger(once=True)
.option("mergeSchema", "true")
.option('checkpointLocation', path+"/bronze_checkpoint")
.toTable('turbine_bronze')) #.start(path + "/turbine_bronze"))
Labels:
- Labels:
-
Delta Lake Files