Hi All,

I am facing an issue of data getting missed.

I am reading the data from azure event hub and after flattening the json data I am storing it in a parquet file and then using another databricks notebook to perform the merge operations on my delta table by adding some etl columns to it.

However in between somewhere the records are getting missed.

I have scheduled the job to run every hour.

Can someone please help me out with this.