cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to write structured streaming

espenol
New Contributor III

Without streaming everything works fine. When I try to set up some example structured streaming in my company's new Databricks workspace (we're trying it out) I can't seem to figure out the correct way to save it/write to storage.

Please see the picture below. I don't think my code is correct. I tried to adapt some code I found. If I include the .table line (in the writeStream) then I get the error message as shown. The error message is so unclear I have no idea what I'm supposed to do. If I comment the .table line, then it will initialize the stream, create the directory in adls gen2 storage but never complete, just stuck at "Initializing stream". I do want the .table line though so I can specify catalog, schema and table name of the table.

We are using Unity Catalog and it works fine for normal non-streaming workloads, reading from the same files as this stream does and writing to the same storage accounts+containers.

code and error message

2 REPLIES 2

Rishabh264
Honored Contributor II

hey you code is correct , and after analyzing the error you got , is i guess you dont have a write access to that storage account or path , you only have the read permission not the write permission

refer this link ,

https://docs.databricks.com/external-data/azure-storage.html

Rishabh Pandey

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi, This can be an issue where Storage Blob Data contributor role is missing. Please refer: https://docs.databricks.com/external-data/azure-storage.html

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!