How to write the delta files for managed table? how can I define the sink
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-04-2022 03:20 AM
I have tried below code to write data in a delta table and save the delta files in a sink. I tried using azure storage as sink but I get error as not enough access, I can confirm that I have enough access to azure storage, however I can run the below code with path as users/ankit.kumar@databricks.com/datadash and it runs , however I am not sure where to find this path physically where can I see the delta files which has been created.
%sql
USE CATALOG hive_metastore;
CREATE DATABASE IF NOT EXISTS demo_db;
USE DATABASE demo_db;
#This gives error as shaded.databricks.org.apache.hadoop.fs.azure.AzureException: #hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This request is not #authorized to perform this operation using this permission.
path = "wasbs://abcstoragecontainer@azlogs.blob.core.windows.net"
(bronzeDF.writeStream
.format('delta')
.outputMode("append")
.trigger(once=True)
.option("mergeSchema", "true")
.option('checkpointLocation', path+"/bronze_checkpoint")
.toTable('turbine_bronze')) #.start(path + "/turbine_bronze"))
#This path works but where is this path located..where can I find it
path = "/Users/ankit.kumar@databricks.com/demo_db"
(bronzeDF.writeStream
.format('delta')
.outputMode("append")
.trigger(once=True)
.option("mergeSchema", "true")
.option('checkpointLocation', path+"/bronze_checkpoint")
.toTable('turbine_bronze')) #.start(path + "/turbine_bronze"))
- Labels:
-
Azure Storage
-
Delta
-
Delta Files
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-12-2022 06:07 AM
It is easier to mount ADLS as a folder as explained here:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-02-2022 09:10 PM
Hi @Ankit Kumar
Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?
We'd love to hear from you.
Thanks!
![](/skins/images/8C2A30E5B696B676846234E4B14F2C7B/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/8C2A30E5B696B676846234E4B14F2C7B/responsive_peak/images/icon_anonymous_message.png)