Hi @Jose Gonzalez​ ,

Thanks for the reply.

Write stream is as follows

dataset.writeStream
                 .format("delta")
                 .outputMode("append")
                 .option("checkpointLocation", "dbfs:/checkpoints_v1/<table_name>")
                 .option("mergeSchema", "true")
                 .table("<table_name>")

The Dataset is created after fetching records from delta tables in a stream and applying the flatMapGroupsWithState API on the records.

Few Observations:

1) The probability of the error occurring is more at the higher loads. If the input rate for the flatMapGroupsWithState API is 12000 records/second, then then the error occurs regularly and generally within 2 hours of the start of the job. For lower loads of 4000 records / seconds, the error occurs infrequently .

2) The size of the snapshot file for which the error occurs is 0 bytes.

Let me know if you require any other information.

Thanks & Regards,

Rohan