Hi,

I have a scenario where writeStream query writes the stream data to bronze location and I have to read from bronze, do some processing and finally write it to silver. I use S3 location for delta tables

But for the very first execution , readStream is giving an exception: Table schema is not set. Write data into it or use CREATE TABLE to set the schema.

My readStream query:

(spark

.readStream.format("delta")

.option("ignoreChanges","true")

.option("initialPosition","latest")

.load("/mnt/tmp/bronze")

)

Is there any option that can be set with readStream query to check if there's any data in delta location and if not , it will wait until the data is available?