Hi All,

I am working on a streaming data processing. As a intial step i have read the data from azure eventhub using readstream. now i want to writestream this into a delta table.

My requirement is, The data should present in external location (adls gen2) and the table should be available in my metastore.

When i tried the below code

Code_Snippet:

ext_table_location = "adls path"

autoloader_df.writeStream \

       .format("delta") \

       .outputMode("append") \

       .option("checkpointLocation", checkpoint_directory) \

       .option("mergeSchema", "true") \

       .option("path",ext_table_location) \

       .table(ext_table_location)

It is failing. Is there any standard approach for streaming data for this kind of scenario?

Thanks in Advance!