cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to write structured streaming

espenol
New Contributor III

Without streaming everything works fine. When I try to set up some example structured streaming in my company's new Databricks workspace (we're trying it out) I can't seem to figure out the correct way to save it/write to storage.

Please see the picture below. I don't think my code is correct. I tried to adapt some code I found. If I include the .table line (in the writeStream) then I get the error message as shown. The error message is so unclear I have no idea what I'm supposed to do. If I comment the .table line, then it will initialize the stream, create the directory in adls gen2 storage but never complete, just stuck at "Initializing stream". I do want the .table line though so I can specify catalog, schema and table name of the table.

We are using Unity Catalog and it works fine for normal non-streaming workloads, reading from the same files as this stream does and writing to the same storage accounts+containers.

code and error message

2 REPLIES 2

Rishabh-Pandey
Esteemed Contributor

hey you code is correct , and after analyzing the error you got , is i guess you dont have a write access to that storage account or path , you only have the read permission not the write permission

refer this link ,

https://docs.databricks.com/external-data/azure-storage.html

Rishabh Pandey

Debayan
Databricks Employee
Databricks Employee

Hi, This can be an issue where Storage Blob Data contributor role is missing. Please refer: https://docs.databricks.com/external-data/azure-storage.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group